Systems and methods for image processing

ABSTRACT

Systems and methods for image processing are provided in the present disclosure. The systems may generate a preliminary image by filtering image data generated by an image acquisition device. The system may generate an intermediate image by performing, based on a first objective function, a first iterative operation on the preliminary image. The first objective function may include a first term associated with a first difference between the intermediate image and the preliminary image, a second term associated with continuity of the intermediate image and a third term associated with sparsity of the intermediate image. The systems may also generate a target image by performing, based on a second objective function, a second iterative operation on the intermediate image. The second objective function may be associated with a system matrix of the image acquisition device and a second difference between the intermediate image and the target image.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Continuation of International Application No.PCT/CN2020/094853, filed on Jun. 8, 2020, the contents of which areincorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to the field of image processing, and inparticular, to systems and methods for reconstructing superresolutionimages that extract spatial resolution beyond the image data generatedby an image acquisition device.

BACKGROUND

Original images generated or collected by an image acquisition device(e.g., a microscope, a telescope, a camera, a webcam) usually haverelatively poor resolution and/or relatively low contrast, due toproblems such as photobleaching, phototoxicity, etc. Image processingtechniques are widely used in image reconstruction to produce imageswith higher qualities. However, conventional image processing techniquesare limited by factors such as signal to noise ratio (SNR), physicaloptics limits, or the like, thereby the reconstructed images usuallyhave artifacts or the reconstruction speed may be relatively low.Therefore, it is desirable to provide systems and methods for efficientimage processing to effectively improve image resolution andreconstruction speed.

SUMMARY

In one aspect of the present disclosure, a method for image processingmay be provided. The method may be implemented on at least one machineeach of which has at least one processor and at least one storagedevice. The method may include generating a preliminary image byfiltering image data generated by an image acquisition device. Themethod may include generating an intermediate image by performing, basedon a first objective function, a first iterative operation on thepreliminary image. The first objective function may include a firstterm, a second term and a third term. The first term may be associatedwith a first difference between the intermediate image and thepreliminary image. The second term may be associated with continuity ofthe intermediate image. The third term may be associated with sparsityof the intermediate image. The method may also include generating atarget image by performing, based on a second objective function, asecond iterative operation on the intermediate image. The secondobjective function may be associated with a system matrix of the imageacquisition device and a second difference between the intermediateimage and the target image.

In another aspect of the present disclosure, a system for imageprocessing is provided. The system may include at least one storagemedium including a set of instructions, and at least one processor incommunication with the at least one storage medium. When executing theset of instructions, the at least one processor may be directed to causethe system to perform following operations. The at least one processormay generate a preliminary image by filtering image data generated by animage acquisition device. The at least one processor may generate anintermediate image by performing, based on a first objective function, afirst iterative operation on the preliminary image. The first objectivefunction may include a first term, a second term and a third term. Thefirst term may be associated with a first difference between theintermediate image and the preliminary image. The second term may beassociated with continuity of the intermediate image. The third term maybe associated with sparsity of the intermediate image. The at least oneprocessor may generate a target image by performing, based on a secondobjective function, a second iterative operation on the intermediateimage. The second objective function may be associated with a systemmatrix of the image acquisition device and a second difference betweenthe intermediate image and the target image.

In another aspect of the present disclosure, a non-transitory computerreadable medium is provided. The non-transitory computer readable mediummay include at least one set of instructions for image processing. Whenexecuted by one or more processors of a computing device, the at leastone set of instructions may cause the computing device to perform amethod. The method may include generating a preliminary image byfiltering image data generated by an image acquisition device. Themethod may include generating an intermediate image by performing, basedon a first objective function, a first iterative operation on thepreliminary image. The first objective function may include a firstterm, a second term and a third term. The first term may be associatedwith a first difference between the intermediate image and thepreliminary image. The second term may be associated with continuity ofthe intermediate image. The third term may be associated with sparsityof the intermediate image. The method may also include generating atarget image by performing, based on a second objective function, asecond iterative operation on the intermediate image. The secondobjective function may be associated with a system matrix of the imageacquisition device and a second difference between the intermediateimage and the target image.

In another aspect of the present disclosure, a system for imageprocessing is provided. The system may include a primary imagegeneration module, a sparse reconstruction module, and a deconvolutionmodule. The primary image generation module may be configured togenerate a preliminary image by filtering image data generated by animage acquisition device. The sparse reconstruction module may beconfigured to generate an intermediate image by performing, based on afirst objective function, a first iterative operation on the preliminaryimage. The first objective function may include a first term, a secondterm and a third term. The first term may be associated with a firstdifference between the intermediate image and the preliminary image. Thesecond term may be associated with continuity of the intermediate image.The third term may be associated with sparsity of the intermediateimage. The deconvolution module may be configured to generate a targetimage by performing, based on a second objective function, a seconditerative operation on the intermediate image. The second objectivefunction may be associated with a system matrix of the image acquisitiondevice and a second difference between the intermediate image and thetarget image.

Additional features will be set forth in part in the description whichfollows, and in part will become apparent to those skilled in the artupon examination of the following and the accompanying drawings or maybe learned by production or operation of the examples. The features ofthe present disclosure may be realized and attained by practice or useof various aspects of the methodologies, instrumentalities, andcombinations set forth in the detailed examples discussed below.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is further described in detail of exemplaryembodiments. These exemplary embodiments are described in detail withreference to the drawings. These embodiments are not limiting, and inthese embodiments, the same number indicates the same structure,wherein:

FIG. 1 is a schematic diagram illustrating an exemplary applicationscenario of an image processing system according to some embodiments ofthe present disclosure;

FIG. 2 is a schematic diagram illustrating an exemplary computing deviceaccording to some embodiments of the present disclosure;

FIG. 3 is a block diagram illustrating an exemplary mobile device onwhich the terminal(s) may be implemented according to some embodimentsof the present disclosure;

FIG. 4 is schematic diagrams illustrating an exemplary processing deviceaccording to some embodiments of the present disclosure;

FIG. 5 is a flowchart illustrating an exemplary process for imageprocessing according to some embodiments of the present disclosure;

FIG. 6 is a flowchart illustrating an exemplary process for backgroundestimation according to some embodiments of the present disclosure;

FIG. 7 illustrates exemplary results indicating a flowchart of thesparse deconvolution algorithm according to some embodiments of thepresent disclosure;

FIGS. 8A-8Q illustrate exemplary results indicating sparse-SIM mayachieve substantially 60 nm and millisecond spatiotemporal resolution inlive cells according to some embodiments of the present disclosure;

FIGS. 9A-9G illustrate exemplary results indicating intricate dynamicswithin the mitochondria and between ER-mitochondria visualized bydual-color Sparse-SIM according to some embodiments of the presentdisclosure;

FIGS. 10A-10L illustrate exemplary results indicating sparse SD-SIMenables three-dimensional, multicolor and sub-90-nm resolution for livecell SR imaging according to some embodiments of the present disclosure;

FIGS. 11A-11K illustrate exemplary results indicating that upsamplingmay enable Sparse SD-SIM to overcome the Nyquist sampling limit toaccomplish multicolor 3D SR imaging in live cells according to someembodiments of the present disclosure;

FIG. 12 illustrates exemplary results indicating the detailed workflowof the sparse deconvolution according to some embodiments of the presentdisclosure;

FIGS. 13A-13D illustrate exemplary results indicating benchmark ofspatial resolution at different steps of sparse deconvolution accordingto the synthetic image according to some embodiments of the presentdisclosure;

FIGS. 14A-14D illustrate exemplary results indicating that contributionsof different steps in sparse deconvolution of synthetic images maycorrupt with different noise extents according to some embodiments ofthe present disclosure;

FIGS. 15A-15F illustrate exemplary results indicating bona fideextension of spatial resolution by the sparse deconvolution algorithmwhen processing real biological structures according to some embodimentsof the present disclosure;

FIGS. 16A and 16B illustrate exemplary results indicating OTFs obtainedby the Fourier transform of fluorescent beads visualized under differentconditions according to some embodiments of the present disclosure;

FIGS. 17A and 17B illustrate exemplary results indicatingreconstructions with only sparsity, only continuity, or both thesparsity and continuity priors according to some embodiments of thepresent disclosure;

FIGS. 18A-18E illustrate exemplary results indicating athree-dimensional image stack of fluorescent beads under SD-SIM andSparse SD-SIM according to some embodiments of the present disclosure;

FIGS. 19A-19D illustrate exemplary results indicating two-colorlive-cell imaging of clathrin and actin by Sparse SD-SIM according tosome embodiments of the present disclosure; and

FIGS. 20A and 20B illustrate exemplary results indicating ER-lysosomecontacts revealed by the sparse SD-SIM according to some embodiments ofthe present disclosure.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth by way of examples in order to provide a thorough understanding ofthe relevant disclosure. However, it should be apparent to those skilledin the art that the present disclosure may be practiced without suchdetails. In other instances, well-known methods, procedures, systems,components, and/or circuitry have been described at a relativelyhigh-level, without detail, in order to avoid unnecessarily obscuringaspects of the present disclosure. Various modifications to thedisclosed embodiments will be readily apparent to those skilled in theart, and the general principles defined herein may be applied to otherembodiments and applications without departing from the spirit and scopeof the present disclosure. Thus, the present disclosure is not limitedto the embodiments shown, but to be accorded the widest scope consistentwith the claims.

The terminology used herein is for the purpose of describing particularexample embodiments only and is not intended to be limiting. As usedherein, the singular forms “a,” “an,” and “the” may be intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprise,”“comprises,” and/or “comprising,” “include,” “includes,” and/or“including,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof. It will be understood that the term “object” and“subject” may be used interchangeably as a reference to a thing thatundergoes an imaging procedure of the present disclosure.

It will be understood that the term “system,” “engine,” “unit,”“module,” and/or “block” used herein are one method to distinguishdifferent components, elements, parts, section or assembly of differentlevel in ascending order. However, the terms may be displaced by anotherexpression if they achieve the same purpose.

Generally, the word “module,” “unit,” or “block,” as used herein, refersto logic embodied in hardware or firmware, or to a collection ofsoftware instructions. A module, a unit, or a block described herein maybe implemented as software and/or hardware and may be stored in any typeof non-transitory computer-readable medium or another storage device. Insome embodiments, a software module/unit/block may be compiled andlinked into an executable program. It will be appreciated that softwaremodules can be callable from other modules/units/blocks or themselves,and/or may be invoked in response to detected events or interrupts.Software modules/units/blocks configured for execution on computingdevices (e.g., processor 210 as illustrated in FIG. 2) may be providedon a computer-readable medium, such as a compact disc, a digital videodisc, a flash drive, a magnetic disc, or any other tangible medium, oras a digital download (and can be originally stored in a compressed orinstallable format that needs installation, decompression, or decryptionprior to execution). Such software code may be stored, partially orfully, on a storage device of the executing computing device, forexecution by the computing device. Software instructions may be embeddedin firmware, such as an EPROM. It will be further appreciated thathardware modules/units/blocks may be included in connected logiccomponents, such as gates and flip-flops, and/or can be included ofprogrammable units, such as programmable gate arrays or processors. Themodules/units/blocks or computing device functionality described hereinmay be implemented as software modules/units/blocks but may berepresented in hardware or firmware. In general, themodules/units/blocks described herein refer to logicalmodules/units/blocks that may be combined with othermodules/units/blocks or divided into sub-modules/sub-units/sub-blocksdespite their physical organization or storage. The description mayapply to a system, an engine, or a portion thereof.

It will be understood that when a unit, engine, module or block isreferred to as being “on,” “connected to,” or “coupled to,” anotherunit, engine, module, or block, it may be directly on, connected orcoupled to, or communicate with the other unit, engine, module, orblock, or an intervening unit, engine, module, or block may be present,unless the context clearly indicates otherwise. As used herein, the term“and/or” includes any and all combinations of one or more of theassociated listed items.

It should be noted that when an operation is described to be performedon an image, the term “image” used herein may refer to a dataset (e.g.,a matrix) that contains values of pixels (pixel values) in the image. Asused herein, a representation of an object (e.g., a person, an organ, acell, or a portion thereof) in an image may be referred to as the objectfor brevity. For instance, a representation of a cell or organelle(e.g., mitochondria, endoplasmic reticulum, centrosome, Golgi apparatus,etc.) in an image may be referred to as the cell or organelle forbrevity. As used herein, an operation on a representation of an objectin an image may be referred to as an operation on the object forbrevity. For instance, a segmentation of a portion of an image includinga representation of a cell or organelle from the image may be referredto as a segmentation of the cell or organelle for brevity.

It should be understood that the term “resolution” as used herein,refers to a measure of the sharpness of an image. The term“superresolution” or “super-resolved” or “SR” as used herein, refers toan enhanced (or increased) resolution, e.g., which may be obtained by aprocess of combining a sequence of low-resolution images to generate ahigher resolution image or sequence. The term “fidelity” or “integrity”as used herein, refers to a degree to which an electronic device (e.g.,an image acquisition device) accurately reproduces its effect (e.g.,image). The term “continuity” as used herein, refers to a feature thatis temporally and/or spatially continuous. The term “sparsity” as usedherein, refers to a feature that a vector or matrix is mostly zeros,e.g., a count of values of 0 in a vector or matrix is much greater thana count of values of 1 in the vector or matrix.

These and other features, and characteristics of the present disclosure,as well as the methods of operation and functions of the relatedelements of structure and the combination of parts and economies ofmanufacture, may become more apparent upon consideration of thefollowing description with reference to the accompanying drawings, allof which form a part of this disclosure. It is to be expresslyunderstood, however, that the drawings are for the purpose ofillustration and description only and are not intended to limit thescope of the present disclosure. It is understood that the drawings arenot to scale.

The flowcharts used in the present disclosure illustrate operations thatsystems implement according to some embodiments of the presentdisclosure. It is to be expressly understood the operations of theflowcharts may be implemented not in order. Conversely, the operationsmay be implemented in inverted order, or simultaneously. Moreover, oneor more other operations may be added to the flowcharts. One or moreoperations may be removed from the flowcharts.

The present disclosure relates to systems and methods for imageprocessing. The systems and methods may generate a preliminary image byfiltering image data generated by an image acquisition device. Thesystems and methods may generate an intermediate image by performing,based on a first objective function, a first iterative operation on thepreliminary image. The first objective function may include a firstterm, a second term, and a third term. The first term may be associatedwith a first difference between the intermediate image and thepreliminary image. The second term may be associated with the continuityof the intermediate image. The third term may be associated with thesparsity of the intermediate image. The systems and methods may generatea target image by performing, based on a second objective function, asecond iterative operation on the intermediate image. The secondobjective function may be associated with a system matrix of the imageacquisition device and a second difference between the intermediateimage and the target image.

According to the systems and methods of the present disclosure, theimage data generated from the image acquisition device may bereconstructed in consideration of the fidelity (and/or integrity), thecontinuity and the sparsity of the image information to generate thetarget image, thereby improving reconstruction speed and reducingphotobleaching. The processed (or reconstructed) target image mayinclude an improved resolution and contrast in comparison to a referenceimage that is generated in consideration of only the fidelity and thecontinuity. What's more, the systems and methods can extend theresolution of the target image beyond physical limits posed by the imageacquisition device, and can permit extrapolations of bandwidth limitedby the image acquisition device.

Moreover, although the systems and methods disclosed in the presentdisclosure are described primarily regarding the processing of imagesgenerated by structured illumination microscopy, it should be understoodthat the descriptions are merely provided for the purposes ofillustration, and not intended to limit the scope of the presentdisclosure. The systems and methods of the present disclosure may beapplied to any other kind of systems including an image acquisitiondevice for image processing. For example, the systems and methods of thepresent disclosure may be applied to microscopes, telescopes, cameras(e.g., surveillance cameras, camera phones, wedcams), unmanned aerialvehicles, medical imaging devices, or the like, or any combinationthereof.

Merely by way of example, the methods disclosed in the presentdisclosure may be used to process images generated by structuredillumination microscopy (SIM), and obtain reconstructed images with arelatively high resolution. In some embodiments, to temporally encodesuperresolution information via specific optics and fluorescent on-offindicators, modern live-cell superresolution microscopes may beultimately limited by the maximum collected photon flux. Takingadvantage of a priori knowledge of the sparsity and continuity offluorescent structures, a mathematical deconvolution algorithm thatextends resolution beyond physical limits posed by the hardware may beprovided in the present disclosure. As a result, sparse structuredillumination microscopy (Sparse-SIM) (e.g., SIM using sparsereconstruction (e.g., sparse deconvolution) as described in FIG. 5) mayachieve substantially 60 nm resolution at a temporal resolvability ofsubstantially 2 ms, allowing it to resolve intricate structures (e.g.,small vesicular fusion pores, relative movements between the inner andouter membranes of mitochondria in live cells, etc.). Likewise, sparsespinning-disc confocal-based SIM may allow routine four-color andthree-dimensional live-cell SR imaging (with substantially 90 nmresolution), demonstrating the general applicability of sparsedeconvolution.

The emergence of superresolution (SR) fluorescence microscopytechnologies may have revolutionized biology and enabled previouslyunappreciated, intricate structures to be observed, such as periodicactin rings in neuronal dendrites, nuclear pore complex structures, andthe organization of pericentriolar materials surrounding the centrioles.However, many of these earlier experiments were conducted in fixed cellsin which the dynamic structural and functional changes of cellularstructures were lost.

Despite theoretically unlimited resolution for all major classes of SRtechnologies, the spatial resolution of live-cell SR microscopy may belimited. On the one hand, for a finite number of fluorophores within thecell volume, an N-fold increase in spatial resolution in the XYZ axesmay lead to voxels that are N³-fold smaller, requiring a more thanN³-fold increase in illumination intensity to maintain the samecontrast. Furthermore, because multiple raw images are usually taken toreconstruct one super-resolved image, relatively small voxels may bemore likely to be affected by motion artifacts of mobile molecules inlive cells, which may degrade the achievable resolution. Therefore, anyincrease in spatial resolution may need to be matched with an increasein temporal resolution to maintain meaningful resolution. Given thesaturation of fluorescence emissions at excessive illumination, thehighest spatial resolution of current live-SR microscopy may be limitedto, e.g., substantially 60 nm, irrespective of the imaging modalitiesused. To achieve that resolution, excessive illumination power (e.g.,kW-MW/mm²) and/or long exposures (e.g., >2 s) may be usually required,which may compromise both the integrity of the holistic fluorescentstructure and the practical resolution of fast-moving subcellularstructures, such as tubular endoplasmic reticulum, lipid droplets,mitochondria, and lysosomes.

Compared to other types of SR imaging technologies, structuredillumination microscopy (SIM) may have higher photon efficiency but maybe prone to artifacts. By developing a Hessian deconvolution for SIMreconstruction, high-fidelity SR imaging may be achieved using onlyone-tenth of the illumination dose needed in conventional SIM, whichenables ultrafast and hour-long imaging in live cells. However, HessianSIM may suffer from a resolution limit of, e.g., 90-110 nm posed bylinear spatial frequency mixing, which prevents structures such as smallcaveolae or vesicle fusion pores from being well resolved. Althoughnonlinear SIM may be used to achieve substantially 60 nm resolution,such implementations may come at a cost of reduced temporal resolutionand may require photoactivatable or photoswitchable fluorescent proteinsthat are susceptible to photobleaching. In addition, becauseconventional SIM uses wide-field illumination, the fluorescenceemissions and the scattering from the out-of-focus planes may compromiseimage contrast and resolution inside deep layers of the cell cytosol andnucleus. Although total internal reflection illumination or grazingincidence illumination can be combined with SIM, high-contrast SR-SIMimaging may be largely limited to imaging depths of, e.g., 0.1-1 μm. Itmay be challenging for SR technologies to achieve millisecond framerates with substantially 60 nm spatiotemporal resolution (e.g., in livecells), or be capable of multiple-color, three-dimensional, and/orlong-term SR imaging.

As first proposed in the 1960s and 1970s, mathematical bandwidthextrapolation may be used to boost spatial resolution. It follows thatwhen the object been imaged has a finite size, there may exist a uniqueanalytic function that coincides inside the bandwidth-limited frequencyspectrum band of the optical transfer function (OTF) of the imageacquisition device, thus enabling the reconstruction of the completeobject by extrapolating the observed spectrum. However, these bandwidthextrapolation operations may fail in actual applications because thereconstruction critically depends on the accuracy and availability ofthe assumed a priori knowledge. By replacing specific prior knowledgeregarding the object itself with a more general sparsity assumption, acompressive sensing paradigm may enable SR in proof-of-principleexperiments. However, such reconstructions may be unstable unless themeasurement is carried out in the near field, while its resolution limitis inversely proportional to the SNR. Thus, although theoreticallypossible, it may be challenging to demonstrate mathematical SRtechnologies in real biological experiments. More descriptions of themathematical SR technologies may be found elsewhere in the presentdisclosure (e.g., FIGS. 4-6 and the description thereof). Moredescriptions of the biological experiments may be found elsewhere in thepresent disclosure (e.g., Examples. 1-5, FIGS. 7-20 and descriptionsthereof).

By incorporating both the sparsity and the continuity priors, adeconvolution algorithm that can surpass the physical limitationsimposed by the hardware of the image acquisition device and increaseboth the resolution and contrast of live cell SR microscopy images maybe provided. By applying the sparse deconvolution algorithm towide-field microscopy, SIM, and spinning-disc confocal-based SIM(SD-SIM), legitimate mathematical SR that allows intricate sub-cellularstructures and dynamics to be visualized in live cells may bedemonstrated.

It should be understood that application scenarios of systems andmethods disclosed herein are only some exemplary embodiments providedfor the purposes of illustration, and not intended to limit the scope ofthe present disclosure. For persons having ordinary skills in the art,multiple variations and modifications may be made under the teachings ofthe present disclosure.

FIG. 1 is a schematic diagram illustrating an exemplary applicationscenario of an image processing system according to some embodiments ofthe present disclosure. As shown in FIG. 1, the image processing system100 may include an image acquisition device 110, a network 120, one ormore terminals 130, a processing device 140, and a storage device 150.

The components in the image processing system 100 may be connected inone or more of various ways. Merely by way of example, the imageacquisition device 110 may be connected to the processing device 140through the network 120. As another example, the image acquisitiondevice 110 may be connected to the processing device 140 directly asindicated by the bi-directional arrow in dotted lines linking the imageacquisition device 110 and the processing device 140. As still anotherexample, the storage device 150 may be connected to the processingdevice 140 directly or through the network 120. As a further example,the terminal 130 may be connected to the processing device 140 directly(as indicated by the bi-directional arrow in dotted lines linking theterminal 130 and the processing device 140) or through the network 120.

The imaging processing system 100 may be configured to generate a targetimage using the sparse deconvolution technique (e.g., as shown in FIG.5). The target image may be with a relatively high resolution that canextend beyond physical limits posed by the image acquisition device 110.A plurality of target images in a time sequence may form a video streamwith a relatively high resolution. For example, the imaging processingsystem 100 may obtain a plurality of raw cell images with relatively lowresolutions generated by the image acquisition device 110 (e.g., SIM).The imaging processing system 100 may generate an SR cell image based onthe plurality of raw cell images using the sparse deconvolutiontechnique. As another example, the image processing system 100 mayobtain one or more images captured by the image acquisition device 110(e.g., a camera phone or a phone with a camera). The one or more imagesmay be blurred and/or with relatively low resolutions, as factors suchas a shaking of the camera phone, moving of an object to be imaged, aninaccurate focusing, etc. during the capturing and/or physical limitsposed by the camera phone. The image processing system 100 may processthe image(s) by using the sparse deconvolution technique to generate oneor more target images with relatively high resolutions. The processedtarget image (s) may be sharper than the image(s) without beingprocessed by the sparse deconvolution technique. Thus, the imageprocessing system 100 may display the target images(s) with a relativelyhigh resolution for a user of the image processing system 100.

The image acquisition device 110 may be configured to obtain image dataassociated with a subject within its detection region. In the presentdisclosure, “object” and “subject” are used interchangeably. The subjectmay include one or more biological or non-biological objects. In someembodiments, the image acquisition device 110 may be an optical imagingdevice, a radioactive-ray-based imaging device (e.g., a computedtomography device), a nuclide-based imaging device (e.g., a positronemission tomography device), a magnetic resonance imaging device), etc.Exemplary optical imaging devices may include a microscope 111 (e.g., afluorescence microscope), a surveillance device 112 (e.g., a securitycamera), a mobile terminal device 113 (e.g., a camera phone), a scanningdevice 114 (e.g., a flatbed scanner, a drum scanner, etc.), a telescope,a webcam, or the like, or any combination thereof. In some embodiments,the optical imaging device may include a capture device (e.g., adetector or a camera) for collecting the image data. For illustrationpurposes, the present disclosure may take the microscope 111 as anexample for describing exemplary functions of the image acquisitiondevice 110. Exemplary microscopes may include a structured illuminationmicroscope (SIM) (e.g., a two-dimensional SIM (2D-SIM), a three-dimensional SIM (3D-SIM), a total internal reflection SIM (TIRF-SIM), aspinning-disc confocal-based SIM (SD-SIM), etc.), a photoactivatedlocalization microscopy (PALM), a stimulated emission depletionfluorescence microscopy (STED), a stochastic optical reconstructionmicroscopy (STORM), etc. The SIM may include a detector such as an EMCCDcamera, an sCMOS camera, etc. The subjects detected by the SIM mayinclude one or more objects of biological structures, biological issues,proteins, cells, microorganisms, or the like, or any combination.Exemplary cells may include INS-1 cells, COS-7 cells, Hela cells, liversinusoidal endothelial cells (LSECs), human umbilical vein endothelialcells (HUVECs), HEK293 cells, or the like, or any combination thereof.In some embodiments, the one or more objects may be fluorescent orfluorescent-labeled. The fluorescent or fluorescent-labeled objects maybe excited to emit fluorescence for imaging. More descriptions of anexemplary SIM device may be found in Chinese Patent ApplicationNo.201810196491.X entitled “A METHOD AND SYSTEM FOR IMAGERECONSTRUCTION.” filed Mar. 9, 2018, the contents of which are herebyincorporated by reference.

The network 120 may include any suitable network that can facilitate theimage processing system 100 to exchange information and/or data. In someembodiments, one or more of components (e.g., the image acquisitiondevice 110, the terminal(s) 130, the processing device 140, the storagedevice 150, etc.) of the image processing system 100 may communicateinformation and/or data with one another via the network 120. Forexample, the processing device 140 may acquire image data from the imageacquisition device 110 via the network 120. As another example, theprocessing device 140 may obtain user instructions from the terminal(s)130 via the network 120. The network 120 may be and/or include a publicnetwork(e.g., the Internet), a private network (e.g., a local areanetwork (LAN), a wide area network (WAN), etc.), a wired network (e.g.,an Ethernet), a wireless network (e.g., an 802.11 network, a Wi-Finetwork, etc.), a cellular network (e.g., a Long Term Evolution (LTE)network), an image relay network, a virtual private network (“VPN”), asatellite network, a telephone network, a router, a hub, a switch, aserver computer, and/or a combination of one or more thereof. Forexample, the network 120 may include a cable network, a wired network, afiber network, a telecommunication network, a local area network, awireless local area network (WLAN), a metropolitan area network (MAN), apublic switched telephone network (PSTN), a Bluetooth™ network, aZigBee™ network, a near field communication network (NFC), or the like,or a combination thereof. In some embodiments, the network 120 mayinclude one or more network access points. For example, the network 120may include wired and/or wireless network access points, such as basestations and/or network switching points, through which one or morecomponents of the image processing system 100 may access the network 120for data and/or information exchange.

In some embodiments, a user may operate the image processing system 100through the terminal(s) 130. The terminal(s) 130 may include a mobiledevice 131, a tablet computer 132, a laptop computer 133, or the like,or a combination thereof. In some embodiments, the mobile device 131 mayinclude a smart home device, a wearable device, a mobile device, avirtual reality device, an augmented reality device, or the like. Insome embodiments, the smart home device may include a smart lightingdevice, a control device of an intelligent electrical apparatus, a smartmonitoring device, a smart television, a smart video camera, aninterphone, or the like, or a combination thereof. In some embodiments,the wearable device may include a bracelet, footgear, glasses, a helmet,a watch, clothing, a backpack, a smart accessory, or the like, or acombination thereof. In some embodiments, the mobile device may includea mobile phone, a personal digital assistant (PDA), a gaming device, anavigation device, a point of sale (POS) device, a laptop, a tabletcomputer, a desktop, or the like, or a combination thereof. In someembodiments, the virtual reality device and/or augmented reality devicemay include a virtual reality helmet, virtual reality glasses, a virtualreality eyewear, an augmented reality helmet, augmented reality glasses,an augmented reality eyewear, or the like, or a combination thereof. Forexample, the virtual reality device and/or augmented reality device mayinclude a Google Glass™, an Oculus Rift™, a Hololens™, a Gear VR™, orthe like. In some embodiments, the terminal(s) 130 may be part of theprocessing device 140.

The processing device 140 may process data and/or information obtainedfrom the image acquisition device 110, the terminal(s) 130, and/or thestorage device 150. For example, the processing device 140 may processimage data generated by the image acquisition device 110 to generate atarget image with a relatively high resolution. In some embodiments, theprocessing device 140 may be a server or a server group. The servergroup may be centralized or distributed. In some embodiments, theprocessing device 140 may be local or remote. For example, theprocessing device 140 may access information and/or data stored in theimage acquisition device 110, the terminal(s) 130, and/or the storagedevice 150 via the network 120. As another example, the processingdevice 140 may be directly connected to the image acquisition device110, the terminal(s) 130, and/or the storage device 150 to access storedinformation and/or data. In some embodiments, the processing device 140may be implemented on a cloud platform. For example, the cloud platformmay include a private cloud, a public cloud, a hybrid cloud, a communitycloud, a distributed cloud, an interconnected cloud, a multiple cloud,or the like, or a combination thereof. In some embodiments, theprocessing device 140 may be implemented by a computing device 200having one or more components as described in FIG. 2.

The storage device 150 may store data, instructions, and/or any otherinformation. In some embodiments, the storage device 150 may store dataobtained from the terminal(s) 130, the image acquisition device 110,and/or the processing device 140. In some embodiments, the storagedevice 150 may store data and/or instructions that the processing device140 may execute or use to perform exemplary methods described in thepresent disclosure. In some embodiments, the storage device 150 mayinclude a mass storage device, a removable storage device, a volatileread-and-write memory, a read-only memory (ROM), or the like. Exemplarymass storage devices may include a magnetic disk, an optical disk, asolid-state drive, etc. Exemplary removable storage devices may includea flash drive, a floppy disk, an optical disk, a memory card, a zipdisk, a magnetic tape, etc. Exemplary volatile read-and-write memory mayinclude a random access memory (RAM). Exemplary RAM may include adynamic RAM (DRAM), a double date rate synchronous dynamic RAM (DDRSDRAM), a static RAM (SRAM), a thyristor RAM (T-RAM), and azero-capacitor RAM (Z-RAM), etc. Exemplary ROM may include a mask ROM(MROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM),an electrically erasable programmable ROM (EEPROM), a compact disk ROM(CD-ROM), and a digital versatile disk ROM, etc. In some embodiments,the storage device 150 may be executed on a cloud platform. For example,the cloud platform may include a private cloud, a public cloud, a hybridcloud, a community cloud, a distributed cloud, an interconnected cloud,a multiple cloud, or the like, or a combination thereof.

In some embodiments, the storage device 150 may be connected to thenetwork 120 to communicate with one or more other components (e.g., theprocessing device 140, the terminal(s) 130, etc.) of the imageprocessing system 100. One or more components of the image processingsystem 100 may access data or instructions stored in the storage device150 via the network 120. In some embodiments, the storage device 150 maybe directly connected to or communicate with one or more othercomponents (e.g., the processing device 140, the terminal(s) 130, etc.)of the image processing system 100. In some embodiments, the storagedevice 150 may be part of the processing device 140.

FIG. 2 is a schematic diagram illustrating an exemplary computing deviceaccording to some embodiments of the present disclosure. The computingdevice 200 may be used to implement any component of the imageprocessing system 100 as described herein. For example, the processingdevice 140 and/or the terminal(s) 130 may be implemented on thecomputing device 200, respectively, via its hardware, software program,firmware, or a combination thereof. Although only one such computingdevice is shown, for convenience, the computer functions relating to theimage processing system 100 as described herein may be implemented in adistributed manner on a number of similar platforms, to distribute theprocessing load.

As shown in FIG. 2, the computing device 200 may include a processor210, a storage 220, an input/output (I/O) 230, and a communication port240.

The processor 210 may execute computer instructions (e.g., program code)and perform functions of the image processing system 100 (e.g., theprocessing device 140) in accordance with the techniques describedherein. The computer instructions may include, for example, routines,programs, objects, components, data structures, procedures, modules, andfunctions, which perform particular functions described herein. Forexample, the processor 210 may process image data obtained from anycomponents of the image processing system 100. In some embodiments, theprocessor 210 may include one or more hardware processors, such as amicrocontroller, a microprocessor, a reduced instruction set computer(RISC), an application specific integrated circuits (ASICs), anapplication-specific instruction-set processor (ASIP), a centralprocessing unit (CPU), a graphics processing unit (GPU), a physicsprocessing unit (PPU), a microcontroller unit, a digital signalprocessor (DSP), a field programmable gate array (FPGA), an advancedRISC machine (ARM), a programmable logic device (PLD), any circuit orprocessor capable of executing one or more functions, or the like, or acombination thereof.

Merely for illustration, only one processor is described in thecomputing device 200. However, it should be noted that the computingdevice 200 in the present disclosure may also include multipleprocessors, thus operations and/or method operations that are performedby one processor as described in the present disclosure may also bejointly or separately performed by the multiple processors. For example,if in the present disclosure the processor of the computing device 200executes both operation A and operation B, it should be understood thatoperation A and operation B may also be performed by two or moredifferent processors jointly or separately in the computing device 200(e.g., a first processor executes operation A and a second processorexecutes operation B, or the first and second processors jointly executeoperations A and B).

The storage 220 may store data/information obtained from any componentof the image processing system 100. In some embodiments, the storage 220may include a mass storage device, a removable storage device, avolatile read-and-write memory, a read-only memory (ROM), or the like,or any combination thereof. Exemplary mass storage devices may include amagnetic disk, an optical disk, a solid-state drive, etc. The removablestorage device may include a flash drive, a floppy disk, an opticaldisk, a memory card, a zip disk, a magnetic tape, etc. The volatileread-and-write memory may include a random access memory (RAM). The RAMmay include a dynamic RAM (DRAM), a double date rate synchronous dynamicRAM (DDR SDRAM), a static RAM (SRAM), a thyristor RAM (T-RAM), and azero-capacitor RAM (Z-RAM), etc. The ROM may include a mask ROM (MROM),a programmable ROM (PROM), an erasable programmable ROM (EPROM), anelectrically erasable programmable ROM (EEPROM), a compact disk ROM(CD-ROM), and a digital versatile disk ROM, etc.

In some embodiments, the storage 220 may store one or more programsand/or instructions to perform exemplary methods described in thepresent disclosure. For example, the storage 220 may store a program forthe processing device 140 to process images generated by the imageacquisition device 110.

The I/O 230 may input and/or output signals, data, information, etc. Insome embodiments, the I/O 230 may enable user interaction with the imageprocessing system 100 (e.g., the processing device 140). In someembodiments, the I/O 230 may include an input device and an outputdevice. Examples of the input device may include a keyboard, a mouse, atouch screen, a microphone, or the like, or a combination thereof.Examples of the output device may include a display device, aloudspeaker, a printer, a projector, or the like, or a combinationthereof. Examples of the display device may include a liquid crystaldisplay (LCD), a light-emitting diode (LED)-based display, a flat paneldisplay, a curved screen, a television device, a cathode ray tube (CRT),a touch screen, or the like, or a combination thereof.

The communication port 240 may be connected to a network to facilitatedata communications. The communication port 240 may establishconnections between the processing device 140 and the image acquisitiondevice 110, the terminal(s) 130, and/or the storage device 150. Theconnection may be a wired connection, a wireless connection, any othercommunication connection that can enable data transmission and/orreception, and/or any combination of these connections. The wiredconnection may include, for example, an electrical cable, an opticalcable, a telephone wire, or the like, or a combination thereof. Thewireless connection may include a Bluetooth™ link, a Wi-Fi™ link, aWiMax™ link, a WLAN link, a ZigBee™ link, a mobile network link (e.g.,3G, 4G, 5G), or the like, or a combination thereof. In some embodiments,the communication port 240 may be and/or include a standardizedcommunication port, such as RS232, RS485, etc. In some embodiments, thecommunication port 240 may be a specially designed communication port.For example, the communication port 240 may be designed in accordancewith the digital imaging and communications in medicine (DICOM)protocol.

FIG. 3 is a block diagram illustrating an exemplary mobile device onwhich the terminal(s) 130 may be implemented according to someembodiments of the present disclosure.

As shown in FIG. 3, the mobile device 300 may include a communicationunit 310, a display unit 320, a graphics processing unit (GPU) 330, acentral processing unit (CPU) 340, an I/O 350, a memory 360, a storageunit 370, etc. In some embodiments, any other suitable component,including but not limited to a system bus or a controller (not shown),may also be included in the mobile device 300. In some embodiments, anoperating system 361 (e.g., iOS™, Android™, Windows Phone™, etc.) andone or more applications (apps) 362 may be loaded into the memory 360from the storage unit 370 in order to be executed by the CPU 340. Theapplication(s) 362 may include a browser or any other suitable mobileapps for receiving and rendering information relating to imaging, imageprocessing, or other information from the image processing system 100(e.g., the processing device 140). User interactions with theinformation stream may be achieved via the I/O 350 and provided to theprocessing device 140 and/or other components of the image processingsystem 100 via the network 120. In some embodiments, a user may inputparameters to the image processing system 100, via the mobile device300.

In order to implement various modules, units and their functionsdescribed above, a computer hardware platform may be used as hardwareplatforms of one or more elements (e.g., the processing device 140and/or other components of the image processing system 100 described inFIG. 1). Since these hardware elements, operating systems and programlanguages are common; it may be assumed that persons skilled in the artmay be familiar with these techniques and they may be able to provideinformation needed in the imaging and assessing according to thetechniques described in the present disclosure. A computer with the userinterface may be used as a personal computer (PC), or other types ofworkstations or terminal devices. After being properly programmed, acomputer with the user interface may be used as a server. It may beconsidered that those skilled in the art may also be familiar with suchstructures, programs, or general operations of this type of computingdevice.

FIG. 4 is schematic diagrams illustrating an exemplary processing deviceaccording to some embodiments of the present disclosure. As shown inFIG. 4, the processing device 140 may include an acquisition module 402,a preliminary image generation module 404, a background estimationmodule 406, an upsampling module 408, a sparse reconstruction module410, and a deconvolution module 412.

The acquisition module 402 may be configured to obtain informationand/data from one or more components of the image processing system 100.In some embodiments, the acquisition module 402 may obtain image datafrom the storage device 150 or the image acquisition device 110. As usedherein, the image data may refer to raw data (e.g., one or more rawimages) collected by the image acquisition device 110. More descriptionsregarding obtaining the image data may be found elsewhere in the presentdisclosure (e.g., operation 502 in FIG. 5).

The preliminary image generation module 404 may be configured togenerate a preliminary image. In some embodiments, the preliminary imagegeneration module 404 may determine the preliminary image by performinga filtering operation on the image data. Merely by way example, thepreliminary image generation module 404 may determine the preliminaryimage by performing Wiener filtering on one or more raw images. Moredescriptions regarding generating the preliminary image may be foundelsewhere in the present disclosure (e.g., operation 504 in FIG. 5).

The background estimation module 406 may be configured to generate anestimated background image based on the preliminary image. In someembodiments, the background estimation module 406 may generate theestimated background image by performing an iterative wavelettransformation operation on the preliminary image. For example, thebackground estimation module 406 may decompose the preliminary imagesinto one or more bands in the frequency domain using a wavelet filter.Exemplary wavelet filters may include Daubechies wavelet filters,Biorthogonal wavelet filters, Morlet wavelet filters, Gaussian waveletfilters, Marr wavelet filters, Meyer wavelet filters, Shannon waveletfilters, Battle-Lemarie wavelet filters, or the like, or any combinationthereof. The background estimation module 406 may extract the lowestband in the frequency domain (e.g., the lowest frequency wavelet band)from one or more bands in the frequency domain. The backgroundestimation module 406 may generate an estimated image based on thelowest frequency wavelet band. If the estimated image satisfies a firsttermination condition, the background estimation module 406 maydetermine the estimated image as the estimated background image.Alternatively, if the estimated image does not satisfy the firsttermination condition, the background estimation module 406 may extractthe lowest frequency wavelet band from the estimated image to determinean updated estimated image until the first termination condition issatisfied. More descriptions regarding generating the estimatedbackground image may be found elsewhere in the present disclosure (e.g.,operation 506 in FIG. 5, operations in FIG. 6 and the descriptionsthereof).

The upsampling module 408 may be configured to perform an upsamplingoperation. In some embodiments, the upsampling module 408 may determinea residual image corresponding to the preliminary image. For example,the upsampling module 408 may extract the estimated background imagefrom the preliminary image to determine the residual image. Theupsampling module 408 may generate an upsampled image by performing anupsampling operation on the residual image, e.g., performing a Fourierinterpolation on the residual image. In some embodiments, the upsampledimage may have a greater size than the residual image. More descriptionsregarding generating the upsampled image based on the preliminary imagemay be found elsewhere in the present disclosure (e.g., operation 508 inFIG. 5 and the description thereof).

The sparse reconstruction module 410 may be configured to perform asparse reconstruction operation. In some embodiments, the sparsereconstruction module 410 may generate an intermediate image byperforming, based on a first objective function, a first iterativeoperation on the preliminary image. The first objective function mayinclude a first term (also referred to as a fidelity term), a secondterm (also referred to as a continuity term) and a third term (alsoreferred to as a sparsity term). The first term may be associated with afirst difference between the intermediate image and the preliminaryimage. The second term may be associated with continuity of theintermediate image. The third term may be associated with sparsity ofthe intermediate image. In some embodiments, the sparse reconstructionmodule 410 may determine an initial estimated intermediate image basedon the preliminary image. If the upsampling operation is omitted, theinitial estimated intermediate image may refer to the residual imagedetermined based on the preliminary image. Alternatively, if theupsampling operation is required, the initial estimated intermediateimage may refer to the upsampled image determined based on the residualimage. The sparse reconstruction module 410 may update an estimatedintermediate image by performing, based on the initial estimatedintermediate image, a plurality of iterations of the first objectfunction. In each of the plurality of iterations, the sparsereconstruction module 410 may determine whether a second terminationcondition is satisfied. In response to determining that the secondtermination condition is satisfied, the sparse reconstruction module 410may finalize the intermediate image and designate the estimatedintermediate image in the iteration as the intermediate image. Inresponse to determining that the second termination condition is notsatisfied, the sparse reconstruction module 410 may determine an updatedestimated intermediate image in the iteration. Further, the sparsereconstruction module 410 may perform a next iteration of the firstobject function based on the updated estimated intermediate image. Moredescriptions regarding generating the intermediate image based on thepreliminary image may be found elsewhere in the present disclosure(e.g., operation 510 in FIG. 5 and the description thereof).

The deconvolution module 412 may be configured to perform adeconvolution operation on the intermediate image to determine a targetimage. In some embodiments, the deconvolution module 412 may generatethe target image by performing, based on a second objective function, asecond iterative operation on the intermediate image. The secondobjective function may be associated with a system matrix of the imageacquisition device 110 and a second difference between the intermediateimage and the target image. The second difference may include a thirddifference between the intermediate image and the target imagemultiplied by the system matrix. In some embodiments, the seconditerative operation may include an iterative deconvolution using aLandweber (LW) deconvolution algorithm, a Richardson Lucy (RL)-baseddeconvolution algorithm, or the like, or any combination thereof. Moredescriptions regarding generating the target image may be foundelsewhere in the present disclosure (e.g., operation 512 in FIG. 5 andthe descriptions thereof).

It should be noted that the above description of modules of theprocessing device 140 is merely provided for the purposes ofillustration, and not intended to limit the present disclosure. Forpersons having ordinary skills in the art, the modules may be combinedin various ways or connected with other modules as sub-systems under theteaching of the present disclosure and without departing from theprinciple of the present disclosure. In some embodiments, one or moremodules may be added or omitted in the processing device 140. Forexample, the preliminary image generation module 404 may be omitted andthe acquisition module 402 may directly obtain the preliminary imagefrom one or more components of the image processing system 100 (e.g.,the image acquisition module 110, the storage device 150, etc.). In someembodiments, one or more modules in the processing device 140 may beintegrated into a single module to perform functions of the one or moremodules. For example, the modules 406-410 may be integrated into asingle module.

In some embodiments, image formation of an aberration-free opticalimaging system may be governed by simple propagation models ofelectromagnetic waves interacting with geometrically ideal surfaces oflenses and mirrors of the optical imaging system. Under such conditions,the process may be described as a two-dimensional Fourier transformationin Equation (1):

G(u, v)=H(u, v)F(u, v),   (1)

where G represents the two-dimensional Fourier transform of the recordedimage, F represents the transformation of the intensity distribution ofthe object associated with the image, and H represents the spatialfrequency transfer function (i.e., the OTF) of the optical imagingsystem. Because the total amount of light energy entering any opticalimaging system is limited by a physically limiting pupil or aperture,the OTF may be controlled by the aperture function under aberration-freeconditions. Therefore, the OTF H(u, v) may be calculated from theaperture function a(x, y) by

H(u, v)=∫_(−∞) ^(∞)∫_(−∞) ^(∞) a(x, y)a(x+u, y+v)dxdy.   (2)

Accordingly, the OTF may be the autocorrelation of the aperturefunction, and x and y denote coordinates in the spatial domain, while uand v represent coordinates in the spatial frequency domain. Thetransmission of electromagnetic waves within the aperture may beperfect, with a(x, y)=1, while outside of the aperture, no wave maypropagate and a(x, y)=0. Therefore, the OTF may change to zero outsideof a boundary defined by the autocorrelation of the aperture function.Because no spatial frequency information from the object outside of theOTF support is transmitted to the recording device, the optical imagingsystem itself may always be bandwidth limited.

In some embodiments, to recover the information of the original objectfrom the recorded image, Equation (1) can be rearranged into thefollowing form:

$\begin{matrix}{{F\left( {u,v} \right)} = {\frac{G\left( {u,v} \right)}{H\left( {u,v} \right)}.}} & (3)\end{matrix}$

In the preceding equation, the retrieved object spatial frequency mayexist only where H(u, v)≠0. Outside of the region of OTF support, theobject spatial frequency may be ambiguous, and any finite value for F(u,v) may be consistent with both Equations (1) and (3). Therefore, theremay be infinite numbers of invisible small objects whose Fouriertransforms are zeros inside the OTF of the optical imaging system andthus are detected. This is why diffraction limits the image resolutionand causes the general belief that the information lost due to thecut-off of system OTF is not retrieved by mathematical inversion ordeconvolution processes.

In some embodiments, in optics, the spatial bandwidth of the opticalimaging system may have traditionally been regarded as invariant.However, analogous to communication systems, in which the channelcapacity may be constant, but the temporal bandwidth may be notconstant. In some embodiments, it may be proposed that only the degreesof freedom (DOF) of an optical imaging system is invariant. Acting as acommunication channel that transmits information from the object planeto the image plane, the ultimate limit may be set by the DOF that theoptical imaging system can transmit.

The DOF of any optical field may be a minimal number of independentvariables needed to describe an optical signal. According to Huygens'principle, if there is no absorption, the two-dimensional informationcarried by the light wave may remain invariant during propagation.Therefore, the information capacity of a two-dimensional optical imagingsystem may be analyzed based on Shannon's information theory, followedby later inclusion of the temporal bandwidth term and the final additionof the z-direction, DC, noise, and polarization terms.

The total information capacity of an optical imaging system may bedescribed as follows:

$\begin{matrix}{{N = {\left( {{2L_{x}B_{x}} + 1} \right)\left( {{2L_{y}B_{y}} + 1} \right)\left( {{2L_{z}B_{z}} + 1} \right) \times \left( {{2TB_{T}} + 1} \right)\left( {\log_{2}\left( {1 + \frac{s}{n}} \right)} \right)}},} & (4)\end{matrix}$

where B_(x), B_(y), and B_(z) represent the spatial bandwidths, andL_(x), L_(y), L_(z) represent the dimensions of the field of view in thex, y, and z-axes, respectively. For any given observation time T, B_(t)denotes the temporal bandwidth of the system. In Equation (4), srepresents the average power of the detected signal while n representsthe additive noise power. The factor 2 accounts for the two polarizationstates.

Exemplary current hardware-based SR techniques, including SIM, STED, andPALM/STORM, may rely on the assumption that the object does not varyduring the formation of an SR image. From Equation (4), it is apparentthat the super-resolved spatial information may be encoded within thetemporal sequence of multiple exposures. In this sense, because theincreased spatial resolution comes at the price of reduced temporalresolution, the information capacity of the optical imaging system mayremain unaltered.

In some embodiments, mathematical bandwidth extrapolation may beperformed and may operate under a different principle than the optics.In some embodiments, the fundamental principle may include that thetwo-dimensional Fourier transform of a spatially bound function is ananalytic function in the spatial frequency domain. If an analyticfunction in the spatial frequency domain is known exactly over a finiteinterval, it may be proven that the entire function can be founduniquely by means of analytic continuation. For an optical imagingsystem, it follows that if an object has a finite size, then a uniqueanalytic function may exist that coincides inside G(u, v). Byextrapolating the observed spectrum using algorithms such as theGerchberg algorithm, it may be possible in principle to reconstruct theobject with arbitrary precision.

Therefore, if the object is assumed to be analytic, there may be apossibility of recovering information from outside the diffractionlimit. Thus, analytic continuation may represent one key feature ofmathematical SR (e.g., the addition of a priori information about theobject for use in its SR reconstruction). In light of Equation (4),adding more information about the object may increase the totalinformation capacity of the system, which theoretically may be divertedto improve the spatial resolution.

The validity of this theory may be proven possible using SR underspecific conditions by iterative deconvolution operations. The basicalgorithmic computation may be given by the following equation:

$\begin{matrix}{{f^{n + 1}\left( {x,y} \right)} = {{{f^{n}\left( {x,y} \right)}\left\lbrack {\left( \frac{g\left( {x,y} \right)}{{h\left( {x,y} \right)} \times {f^{n}\left( {x,y} \right)}} \right) \odot {h\left( {x,y} \right)}} \right\rbrack}.}} & (5)\end{matrix}$

For image data g(x, y) and the point spread function (PSF) h(x, y), theobject at (n+1)^(th) iteration may be calculated from its estimation atn^(th) iteration. Different from direct inversion in the spatialfrequency domain by Equation (3), this computational process may beconducted in the spatial domain, avoiding the problem of division bynull outside of the region of OTF support. Thus, extension beyond thediffraction limit may be possible and may occur for two reasons: thenonlinearity of the product of f^(n) with the other quantities in thesquare brackets and the prior information of nonnegative values of theobject.

Despite the theoretical feasibility of mathematical SR, out-of-bandextrapolation may be usually unstable due to small variations in theimage data. The noise that exists in actual experiments may not be ananalytic function, thus allowing no solution to the bandwidthextrapolation based on analytic continuation. Alternatively, even underconditions where a possible solution exists, there may be many analyticfunctions that reproduce the data well within the OTF but that result incompletely different frequency components outside of OTF.

From the perspective of information capacity, bandwidth extrapolationbased on analytic continuation may extend the spatial bandwidth at theprice of the SNR. For object fields containing large space-bandwidthproducts, even a small extension of spatial bandwidth may lead togreatly amplified noise in the final reconstruction. For object fieldswith small space-bandwidth products, a meaningful increase in resolutionmay be attainable by analytic continuation in combination with theaddition of more object information. Despite all these theoreticalpossibilities, because reconstruction by bandwidth extrapolation isprone to noise and strongly depends on the assumed a priori knowledge,it may be concluded that the Rayleigh diffraction limit represents apractical frontier that cannot be overcome with a conventional imagingsystem.

In some embodiments, compressive sensing (CS), which is performed basedon the sparsity feature of discrete signals, may have been appliedsuccessfully in many different fields of signal processing. CS may be afamily of approaches by which accurate signal reconstruction is possiblewith fewer measurements than that required by the Nyquist-Shannonsampling theorem. CS theory may assume that a discrete signal has asparse representation on some basis. If the measurement is conductedusing a basis that is uncorrelated or incoherent with the signal basis,it may be possible to reconstruct the signal with a sampling rate lessthan twice the maximum frequency of the signal. CS may be combined withpoint scanning confocal microscopy and may be used in pointillist SRtechniques such as PALM and STORM. In some embodiments, applying CStechniques to these imaging technologies may reduce the imageacquisition time and increases the frame rate.

In addition to applying CS for interpolation within the bandwidth, itmay be proposed using sparsity prior for bandwidth extrapolation andmathematical SR. Because the spatial and frequency domains are maximallyuncorrelated bases and the object is measured in the spatial domain, theband-limited frequency information may be extended under the assumptionof sparse signals in the spatial domain. It may be demonstrated that thebandwidth may be extrapolated under the CS framework and more accuratereconstructions may be generated from raw images with some noise thanusing the Gerchberg algorithm. This may provide a proof-of-principlewhere the object is very sparse and no subwavelength resolution isshown. In some embodiments, because noise in the raw images issignificantly amplified during the reconstruction, the subwavelengthmodes may be shown to be typically unstable unless the measurement isconducted in the near field. Likewise, the resolution may be inverselyproportional to the SNR, which limits the possible application of the CStechnique.

In some embodiments, the above mentioned SR technologies based on the CStechnique (e.g., using the sparsity prior) may be combined with thecontinuity prior for image processing, which may keep a balance betweenthe resolution and the SNR to improve the resolution of image data.Details regarding a mathematical SR technology in combination with thecontinuity prior and the sparsity prior may be described as follows(e.g., FIGS. 5-6 and the descriptions thereof).

FIG. 5 is a flowchart illustrating an exemplary process for imageprocessing according to some embodiments of the present disclosure. Insome embodiments, process 500 may be executed by the image processingsystem 100. For example, the process 500 may be implemented as a set ofinstructions (e.g., an application) stored in a storage device (e.g.,the storage device 150, the storage 220, and/or the storage unit 370).In some embodiments, the processing device 140 (e.g., the processor 210of the computing device 200, the CPU 340 of the mobile device 300,and/or one or more modules illustrated in FIG. 4) may execute the set ofinstructions and may accordingly be directed to perform the process 500.The operations of the illustrated process presented below are intendedto be illustrative. In some embodiments, the process may be accomplishedwith one or more additional operations not described, and/or without oneor more of the operations discussed. Additionally, the order in whichthe operations of the process as illustrated in FIG. 5 and describedbelow is not intended to be limiting.

In some embodiments, two criteria may need to be met to increase theresolution mathematically. First, a priori assumption may addinformation to the otherwise invariant total information carried by theoptical imaging system. In this sense, the sparsity prior may be moregeneral than the assumption of object compactness, and the incorporationof more available information regarding the object in the reconstructionalgorithm may lead to greater bandwidth extension. Second, amathematical SR image reconstruction may increase spatial resolution atthe price of a reduction in the SNR following a logarithmicrelationship, hinting at the crucial importance of starting withhigh-SNR images before the bandwidth extrapolation procedure.

To address these issues, a three-step procedure may be provided in thepresent disclosure. In the first step, the SR spatial frequency encodedwithin structured illuminated (grids or pinholes) in the spatialfrequency domain may be estimated, extracted, and reassembled and aninverse filter (e.g., the conventional Wiener inverse filter) may beused to obtain SR images. The first step may boost the SNR of thehigh-frequency components within the bandwidth determined by the optics.The background in the SR images may also be estimated and the SR imagesmay be removed with background fluorescence originating fromout-of-focus planes from the SR images, which may be critical for thehigh-fidelity reconstruction of objects inside the cell. In the secondstep, the continuity prior information may be used to further enhancethe image SNR and the sparsity prior information may be used to extendthe bandwidth to regions out of the OTF. The application of stringentrequirements on sparsity to obtain more accurate reconstruction may leadto the removal of weak bona fide signals. Therefore, in contrast toclassical bandwidth extrapolation techniques that tried to reveal allthe details of an object based on the analytic continuation or CS prioriinformation, the image reconstruction may be framed as an iterativeprocess constrained by the a priori information.

Because of the logarithmic relationship between the resolution increaseand the image SNR, the extrapolated spatial frequency may usually havelow contrast. Therefore, in a third step, an iterative deconvolutionprocedure may be used to boost the contrast of the newly extendedspatial frequency outside of the OTF. The incorporation of thenonnegative priors and the nonlinearity of the deconvolution proceduremay also contribute to the resolution increase to some extent. Moredescriptions regarding the three-step procedure may be found in one ormore operations (e.g., operations 506, 510 and 512) in process 500.

In 502, the processing device 140 (e.g., the acquisition module 402) mayobtain image data.

In some embodiments, the image data herein may refer to raw data (e.g.,one or more raw images) collected by the image acquisition device 110.As used herein, a raw image may refer to an image that has a relativelylow signal to noise ratio (SNR), or is partially damaged, or the like.Merely by way of example, for a SIM system, the image data may includeone or more sets of raw images collected by the SIM system. The one ormore sets of raw images may be arranged in a time sequence. Each set ofraw images may include a plurality of raw images (e.g., 9 raw images, 15raw images) with different phases and/or directions. That is, theplurality of raw images may be collected by the SIM system at thedifferent phases and/or directions.

In some embodiments, the processing device 140 may obtain the image datafrom one or more components of the image processing system 100. Forexample, the image acquisition device 110 may collect and/or generatethe image data and store the image data in the storage device 150. Theprocessing device 140 may retrieve and/or obtain the image data from thestorage device 150. As another example, the processing device 140 maydirectly obtain the image data from the image acquisition device 110.

In 504, the processing device 140 (e.g., the preliminary imagegeneration module 404) may generate a preliminary image based on theimage data.

In some embodiments, the processing device 140 may determine thepreliminary image (e.g., the SR image) by filtering the image data.Exemplary filtering operations may include Wiener filtering, inversefiltering, least-squares filtering, or the like, or any combinationthereof. For example, for each set of raw images of the image data, theprocessing device 140 may generate an image stack (i.e., the preliminaryimage) by performing Wiener filtering on the plurality of raw images inthe set of raw images. Specifically, if each set of raw images includes9 raw images, the processing device 140 may combine the 9 raw images inthe set of raw images into the preliminary image. The preliminary imagemay include information in each of the 9 raw images and have a higherresolution than each of the 9 raw images. In some embodiments, thefiltering operation may be omitted. For example, for a camera phonesystem, the image data may include only one raw image, and theprocessing device 120 may designate the only one raw image as thepreliminary image.

In 506, the processing device 140 (e.g., the background estimationmodule 406) may generate an estimated background image based on thepreliminary image. As used herein, the estimated background image mayrefer to an image including an estimated background (e.g., noise inlow-frequency) of the preliminary image. For example, if objects in thepreliminary image are live cells, the liquid around the live cells inthe preliminary image may be regarded as a noise in the preliminaryimage. The processing device 140 may generate the estimated backgroundimage by estimating the noise in the preliminary image. As anotherexample, for a 2D-SIM system, the preliminary image may contain a strongbackground fluorescence signal originating from out-of-focus emissionsand cellular autofluorescence, which may be regarded as a noise in thepreliminary image. The processing device 140 may predict the noise inthe estimated background image.

In some embodiments, the processing device 140 may generate theestimated background image by performing an iterative wavelettransformation operation on the preliminary image. For example, theprocessing device 140 may decompose the preliminary images into one ormore bands in the frequency domain using a wavelet filter. Exemplarywavelet filters may include Daubechies wavelet filters, Biorthogonalwavelet filters, Morlet wavelet filters, Gaussian wavelet filters, Marrwavelet filters, Meyer wavelet filters, Shannon wavelet filters,Battle-Lemarie wavelet filters, or the like, or any combination thereof.The processing device 140 may extract the lowest band in the frequencydomain (e.g., the lowest frequency wavelet band) from one or more bandsin the frequency domain. The processing device 140 may generate anestimated image based on the lowest frequency wavelet band. If theestimated image satisfies a first termination condition, the processingdevice 140 may determine the estimated image as the estimated backgroundimage. Alternatively, if the estimated image does not satisfy the firsttermination condition, the processing device 140 may extract the lowestfrequency wavelet band from the estimated image to determine an updatedestimated image until the first termination condition is satisfied. Moredescriptions regarding generating the estimated background image may befound elsewhere in the present disclosure (e.g., FIG. 6 and thedescription thereof).

In some embodiments, in a specific condition (e.g., under conditions ofultrafast or long-term 2D-SIM imaging) where the background noise in thepreliminary image is low, the processing device 140 may directlydesignate pixel values in the estimated background image as zero.

In 508, the processing device 140 (e.g., the upsampling module 408) maygenerate an upsampled image based on the preliminary image.

In some embodiments, the processing device 140 may determine a residualimage corresponding to the preliminary image. For example, theprocessing device 140 may determine the residual image by subtractingthe estimated background image from the preliminary image. As anotherexample, when the preliminary image (e.g., the preliminary image in anSD-SIM system) exhibits a low and stable noise-like distribution ofbackground fluorescence only, the processing device 140 may determine amean value of total pixel values in the preliminary image. Theprocessing device 140 may determine the residual image by designatingpixel values in the preliminary image that is less than the mean valueas zero. Further, the processing device 140 may generate the upsampledimage by performing an upsampling operation on the residual image, e.g.,performing a Fourier interpolation on the residual image (e.g., padzeros out of the OTF of the image acquisition device 110). For example,the processing device 140 may pad zeros around one or more pixels (e.g.,each pixel, or pixels on the edge) in the residual image in the spatialdomain to determine the upsampled image. As another example, theprocessing device 140 may pad zeros in the residual image in thefrequency domain to determine the upsampled image. As still anotherexample, for the 2D-SIM system, the processing device 140 may use theFourier interpolation (e.g., pad the zeros out of SIM OTF) to upsamplethe residual image. As further another example, for the SD-SIM system,the processing device 140 may choose a spatial oversampling manner(e.g., pad zeros around each pixel in the residual image), which is lesssensitive to the SNR and less prone to snowflake artifacts.

In some embodiments, the upsampled image may include a larger size thanthe preliminary image. For example, the preliminary image may include asize of 256×256, and the upsampled image may include a size of 512×512.

In some embodiments, the pixel size may be a factor that limits SR imagereconstruction/processing. For example, the pixel size of a 2D-SIMsystem may be 32.5 nm and theoretically be extended to substantially 60nm, which may confer an undersampling problem that limits the spatialresolution. Likewise, when an EMCCD camera is used for the SD-SIMsystem, the pixel size may be 94 nm, which may limit the resolution thatmay be achieved by the sparse deconvolution algorithm (to substantially90 nm). To address the potential problem of undersampling, anoversampling operation (i.e., the upsampling operation) may be providedin the image reconstruction process. In some embodiments, the upsamplingoperation may be omitted and the residual image may be directly used forimage reconstruction.

In 510, the processing device 140 (e.g., the sparse reconstructionmodule 410) may generate an intermediate image by performing, based on afirst objective function, a first iterative operation on the preliminaryimage. The intermediate image may be regarded as an image recovered fromthe preliminary image, such that the intermediate image may also bereferred to as a recovered image. The first iterative operation may alsobe referred to as sparse reconstruction or sparse deconvolution.

In some embodiments, the processing device 140 may determine an initialestimated intermediate image based on the preliminary image. If theupsampling operation is omitted, the initial estimated intermediateimage may refer to the residual image as described in operation 508.Alternatively, if the upsampling operation is required, the initialestimated intermediate image may refer to the upsampled image asdescribed in operation 508. The processing device 140 may update anestimated intermediate image by performing, based on the initialestimated intermediate image, a plurality of iterations of the firstobject function. In each of the plurality of iterations, the processingdevice 140 may determine whether a second termination condition issatisfied. In response to determining that the second terminationcondition is satisfied, the processing device 140 may finalize theintermediate image and designate the estimated intermediate image in theiteration as the intermediate image. In response to determining that thesecond termination condition is not satisfied, the processing device 140may determine an updated estimated intermediate image in the iteration.The processing device 140 may perform a next iteration of the firstobject function based on the updated estimated intermediate image.

In some embodiments, the second termination condition may relate to avalue of the first object function. For example, the second terminationcondition may be satisfied if the value of the first object function isminimal or smaller than a threshold (e.g., a constant). As anotherexample, the second termination condition may be satisfied if the valueof the first object function converges. In some embodiments, convergencemay be deemed to have occurred if the variation of the values of thefirst object function in two or more consecutive iterations is equal toor smaller than a threshold (e.g., a constant). In some embodiments,convergence may be deemed to have occurred if a difference between thevalue of the first object function (e.g., the value of the first objectfunction) and a target value is equal to or smaller than a threshold(e.g., a constant). In some embodiments, the second terminationcondition may relate to an iterative number (count) of the first objectfunction. For example, the second termination condition may be satisfiedwhen a specified iterative number (or count) is performed in the firstiterative operation. In some embodiments, the second terminationcondition may relate to an iterative time of the first object function(e.g., a time length that the first iterative operation is performed).For example, the termination condition may be satisfied if the iterativetime of the first object function is greater than a threshold (e.g., aconstant).

In some embodiments, the first objective function may include a firstterm (also referred to as a fidelity term), a second term (also referredto as a continuity term) and a third term (also referred to as asparsity term). The first term may be associated with a first differencebetween the intermediate image and the preliminary image. The secondterm may be associated with continuity of the intermediate image. Thethird term may be associated with sparsity of the intermediate image. Insome embodiments, for the preliminary image determined without theupsampling operation, the first objective function may be expressed as aloss function as shown in Equation (6):

$\begin{matrix}{{\underset{g}{\arg\mspace{14mu}\min}\left\{ {{\frac{\lambda}{2}{{f - b - g}}_{2}^{2}} + {R_{Hessian}(g)} + {\lambda_{L1}{g}_{1}}} \right\}},} & (6)\end{matrix}$

where

$\frac{\lambda}{2}{{f - b - g}}_{2}^{2}$

represents the first term, R_(Hessian)(g) represents the second term,λ_(L1)∥g∥₁ represents the third term. In the first term, f representsthe preliminary image (also referred to as image f for brevity), brepresents the estimated background image (also referred to as image bfor brevity), g represents the intermediate image (also referred to asimage g or variable g for brevity), ∥⋅∥₁ represents a

norm (i.e., an L-1 norm), and λ represents a first weight factorrelating to image fidelity of the intermediate image. In the secondterm, R_(Hessian)(g) represents a Hessian matrix of the intermediateimage. In the third term, ∥⋅∥₂ represents a

norm (i.e., an L-2 norm), and λ_(L1) represents a second weight factorrelating to sparsity of the intermediate image. λ and λ_(L1) may balanceimage fidelity with sparsity.

In some embodiments, the second term may be associated with a structuralcontinuity along xy-t(z) axes of the intermediate image, i.e, continuityalong the x-axis associated with a dimension of the width of theintermediate image, continuity along the y-axis associated with adimension of the height of the intermediate image, and continuity alongthe t-axis (or z-axis) associated with a time dimension of theintermediate image. For example, the second term (i.e., theR_(Hessian)(g)) may be defined as Equation (7):

$\begin{matrix}{\begin{matrix}{{R_{Hessian}(g)} = {\begin{matrix}g_{xx} & g_{xy} & {\sqrt{\lambda_{t}}g_{xt}} \\g_{yx} & g_{yy} & {\sqrt{\lambda_{t}}g_{yt}} \\{\sqrt{\lambda_{t}}g_{tx}} & {\sqrt{\lambda_{t}}g_{ty}} & {\lambda_{t}g_{tt}}\end{matrix}}_{1}} \\{= \begin{matrix}{{g_{xx}}_{1} + {g_{yy}}_{1} + {\lambda_{t}{g_{tt}}_{1}} +} \\{{2{g_{xy}}_{1}} + {2\sqrt{\lambda_{t}}{g_{xt}}_{1}} + {2\sqrt{\lambda_{t}}{g_{yt}}_{1}}}\end{matrix}}\end{matrix},} & (7)\end{matrix}$

where g_(xx) represents gray values of two-order difference of pixels ofthe image g along the x-axis, g_(xy) represents gray values of two-orderdifference of pixels of the image g along the x-axis and the y-axis,g_(xt) represents gray values of two-order difference of pixels of theimage g along the x-axis and the t-axis, g_(yx) represents gray valuesof two-order difference of pixels of the image g along the y-axis andthe x-axis, g_(yy) represents gray values of two-order difference ofpixels of the image g along the y-axis, g_(yt) represents gray values oftwo-order difference of pixels of the image g along the y-axis and thet-axis, g_(tx) represents gray values of two-order difference of pixelsof the image g along the t-axis and the x-axis, g_(ty) represents grayvalues of two-order difference of pixels of the image g along the t-axisand the y-axis, g_(tt) represents gray values of two-order difference ofpixels of the image g along the t-axis, λ_(t) represents aregularization parameter that presents the continuity along the t-axis(which may be turned off after imaging mobile objects at high speed),and ∥⋅∥₁ represents the L-1 norm.

In some embodiments, searching for the best solution to the lossfunction in Equation (6) (i.e., the termination condition of the lossfunction) may be translated into a convex optimization problem (COP).For example, a split Bregman algorithm (also referred to as Bregmansplitting) may be adopted due to its fast convergence speed. UsingBregman splitting, the variable g may be replaced with an intermediatevariable u. Then, the loss function in Equation (6) may be transformedinto a constrained minimization problem as Equation (8):

$\begin{matrix}{{{{\min\limits_{g,u}\left\{ {{\frac{\lambda}{2}{{f - b - g}}_{2}^{2}} + {\varphi(u)}} \right\}} = {\min\limits_{g,u}\begin{Bmatrix}{{\frac{\lambda}{2}{{f - b - g}}_{2}^{2}} + {u_{xx}}_{1} + {u_{yy}}_{1} + {u_{tt}}_{1} +} \\{{u_{xy}}_{1} + {u_{xt}}_{1} + {u_{yt}}_{1} + {u}_{1}}\end{Bmatrix}}},{where}}{{u_{xx} = g_{xx}},{u_{yy} = g_{yy}},{u_{tt} = {\lambda_{t} \cdot g_{zz}}},{u_{xy} = {2 \cdot g_{xy}}},{u_{xt} = {2{\sqrt{\lambda_{t}} \cdot g_{xt}}}},{u_{yt} = {2{\sqrt{\lambda_{t}} \cdot g_{yt}}}},\ {u = {\lambda_{L1} \cdot {g.}}}}} & (8)\end{matrix}$

In some embodiments, by using a Lagrange multiplier algorithm to weaklyenforce the constraints, an unconstrained problem as Equation (9) may beobtained:

$\begin{matrix}{{\min\limits_{g,u}\left\{ {{\frac{\lambda}{2}{{f - b - g}}_{2}^{2}} + {\varphi(u)} + {\frac{\mu}{2} \cdot \begin{bmatrix}\begin{matrix}{{{u_{xx} - g_{xx}}}_{2}^{2} + {{u_{yy} - g_{yy}}}_{2}^{2} +} \\{{{u_{tt} - {\lambda_{t} \cdot g_{tt}}}}_{2}^{2} + {{u_{xy} - {2 \cdot g_{xy}}}}_{2}^{2} +}\end{matrix} \\\begin{matrix}{{{u_{xt} - {2{\sqrt{\lambda_{t}} \cdot g_{xt}}}}}_{2}^{2} + {{u_{yt} - {2{\sqrt{\lambda_{t}} \cdot}}}}} \\{{g_{yt}}_{2}^{2} + {{u - {\lambda_{L\; 1} \cdot g}}}_{2}^{2}}\end{matrix}\end{bmatrix}}} \right\}},} & (9)\end{matrix}$

where μ represents the Lagrange multiplier. Finally, the constraints maybe strictly enforced by applying a simplified Bregman iterationaccording to Equation (10):

$\begin{matrix}{{\min\limits_{g,u}\left\{ {{\frac{\lambda}{2}{{f - b - g}}_{2}^{2}} + {\varphi(u)} + {\frac{\mu}{2} \cdot \begin{bmatrix}\begin{matrix}{{{u_{xx} - g_{xx} - v_{xx}}}_{2}^{2} + {{u_{yy} - g_{yy} - v_{yy}}}_{2}^{2} +} \\{{{u_{tt} - {\lambda_{t} \cdot g_{tt}} - v_{tt}}}_{2}^{2} + {{u_{xy} - {2 \cdot g_{xy}} - v_{xy}}}_{2}^{2} +}\end{matrix} \\{{{u_{xt} - {2{\sqrt{\lambda_{t}} \cdot g_{xt}}} - v_{xt}}}_{2}^{2} + {{u_{yt} - {2{\sqrt{\lambda_{t}} \cdot}}}}} \\{{{g_{yt} - v_{yt}}}_{2}^{2} + {{u - {\lambda_{L\; 1} \cdot g} - v}}_{2}^{2}}\end{bmatrix}}} \right\}},} & (10)\end{matrix}$

where v is used to reduce the computational complexity in the firstiterative operation.

In some embodiments, the iterative minimization procedure may be givenas follows:

$\begin{matrix}{{g^{k + 1} = {\min\limits_{g}\left\{ {{\frac{\lambda}{2}{{f - b - g}}_{2}^{2}} + {\frac{\mu}{2} \cdot \begin{bmatrix}\begin{matrix}{{{u_{xx} - g_{xx} - v_{xx}}}_{2}^{2} + {{u_{yy} - g_{yy} - v_{yy}}}_{2}^{2} +} \\{{{u_{tt} - {\lambda_{t} \cdot g_{tt}} - v_{tt}}}_{2}^{2} + {{u_{xy} - {2 \cdot g_{xy}} - v_{xy}}}_{2}^{2} +}\end{matrix} \\{{{u_{xt} - {2{\sqrt{\lambda_{t}} \cdot g_{xt}}} - v_{xt}}}_{2}^{2} + {{u_{yt} - {2{\sqrt{\lambda_{t}} \cdot}}}}} \\{{{g_{yt} - v_{yt}}}_{2}^{2} + {{u - {\lambda_{L\; 1} \cdot g} - v}}_{2}^{2}}\end{bmatrix}}} \right\}}},} & (11) \\{\mspace{79mu}{{u_{xx}^{k + 1} = {\min\limits_{u_{xx}}\left\{ {{u_{xx}}_{1} + {\frac{\mu}{2} \cdot {{u_{xx} - g_{xx}^{k + 1} - v_{xx}^{k}}}_{2}^{2}}} \right\}}}\mspace{79mu}{u_{yy}^{k + 1} = {\min\limits_{u_{yy}}\left\{ {{u_{yy}}_{1} + {\frac{\mu}{2} \cdot {{u_{yy} - g_{yy}^{k + 1} - v_{yy}^{k}}}_{2}^{2}}} \right\}}}\mspace{79mu}{u_{tt}^{k + 1} = {\min\limits_{u_{tt}}\left\{ {{u_{tt}}_{1} + {\frac{\mu}{2} \cdot {{u_{tt} - {\lambda_{t} \cdot g_{tt}^{k + 1}} - v_{tt}^{k}}}_{2}^{2}}} \right\}}}\mspace{79mu}{u_{xy}^{k + 1} = {\min\limits_{u_{xy}}\left\{ {{u_{xy}}_{1} + {\frac{\mu}{2} \cdot {{u_{xy} - {\lambda_{z} \cdot g_{xy}^{k + 1}} - v_{xy}^{k}}}_{2}^{2}}} \right\}}}\mspace{79mu}{u_{xt}^{k + 1} = {\min\limits_{u_{xt}}\left\{ {{u_{xt}}_{1} + {\frac{\mu}{2} \cdot {{u_{xt} - {2{\sqrt{\lambda_{t}} \cdot g_{xt}^{k + 1}}} - v_{xt}^{k}}}_{2}^{2}}} \right\}}}\mspace{79mu}{u_{yt}^{k + 1} = {\min\limits_{u_{yt}}\left\{ {{u_{yt}}_{1} + {\frac{\mu}{2} \cdot {{u_{yt} - {2{\sqrt{\lambda_{t}} \cdot g_{yt}^{k + 1}}} - v_{yt}^{k}}}_{2}^{2}}} \right\}}}\mspace{79mu}{{u^{k + 1} = {\min\limits_{u}\left\{ {{u}_{1} + {\frac{\mu}{2} \cdot {{u - {\lambda_{L\; 1} \cdot g^{k + 1}} - v^{k}}}_{2}^{2}}} \right\}}},}}} & (12) \\{\mspace{79mu}{{v_{xx}^{k + 1} = {v_{xx}^{k} + g_{xx}^{k} - u_{xx}^{k}}}\mspace{79mu}{v_{yy}^{k + 1} = {v_{yy}^{k} + g_{yy}^{k} - u_{yy}^{k}}}\mspace{79mu}{v_{tt}^{k + 1} = {v_{tt}^{k} + {\lambda_{t} \cdot g_{tt}^{k}} - u_{tt}^{k}}}\mspace{79mu}{v_{xy}^{k + 1} = {v_{xy}^{k} + {2 \cdot g_{xy}^{k}} - u_{xy}^{k}}}\mspace{79mu}{v_{xt}^{k + 1} = {v_{xt}^{k} + {2{\sqrt{\lambda_{t}} \cdot g_{xt}^{k}}} - u_{xt}^{k}}}\mspace{79mu}{v_{yt}^{k + 1} = {v_{yt}^{k} + {2{\sqrt{\lambda_{t}} \cdot g_{yt}^{k}}} - u_{yt}^{k}}}\mspace{79mu}{v^{k + 1} = {v^{k} + {\lambda_{L\; 1} \cdot g^{k}} - {u^{k}.}}}}} & (13)\end{matrix}$

Using Equations (11-13), the COP may be split into three steps (smoothconvex). Then, the final first iterative operation may be expressed asfollows:

$\begin{matrix}{{g^{k + 1} = \frac{{L^{k}}_{sHe} + {\frac{\lambda}{\mu} \cdot \left( {f - b} \right)}}{\begin{matrix}\left| {{{\nabla_{xx}^{2}{+ {\nabla_{y\gamma}^{2}{+ \lambda_{t}}}}} \cdot {\nabla_{tt}^{2}{+ 2}} \cdot {\nabla_{xy}^{2}{+ 2}}}{\sqrt{\lambda_{t}} \cdot}} \right. \\{{{{\nabla_{xt}^{2}{+ 2}}{\sqrt{\lambda_{t}} \cdot \nabla_{yt}^{2}}}}^{2} + \lambda_{L1}^{2} + \frac{\lambda}{\mu}}\end{matrix}}}{{{L^{k}}_{sHe} = \begin{Bmatrix}\begin{matrix}{{\left( \nabla_{xx}^{2} \right)^{T}\left( {u_{xx}^{k} - v_{xx}^{k}} \right)} + {\left( \nabla_{yy}^{2} \right)^{T}\left( {u_{yy}^{k} - v_{yy}^{k}} \right)} + {\lambda_{z} \cdot \left( \nabla_{tt}^{2} \right)^{T}}} \\{\left( {u_{tt}^{k} - v_{tt}^{k}} \right) + {{2 \cdot \left( \nabla_{xy}^{2} \right)^{T}}\left( {u_{xy}^{k} - v_{xy}^{k}} \right)} +}\end{matrix} \\{{2{\sqrt{\lambda_{t}} \cdot \left( \nabla_{xt}^{2} \right)^{T}}\left( {u_{xt}^{k} - v_{xt}^{k}} \right)} + {2{\sqrt{\lambda_{t}} \cdot}}} \\{{\left( \nabla_{yt}^{2} \right)^{T}\left( {u_{yt}^{k} - v_{yt}^{k}} \right)} + {\lambda_{L\; 1}\left( {u^{k} - v^{k}} \right)}}\end{Bmatrix}},}} & (14)\end{matrix}$

where g^(k+1) represents the intermediate image finally determined inthe last iteration, ∇_(xx) represents a two-order derivation operator inthe x-axis direction (e.g., ∇_(xx)=[1, −2, 1]), and ∇_(yy), ∇_(tt),∇_(xy), ∇_(xt), ∇_(yt) may be similarly defined as ∇_(xx). Arelationship between u^(k+1) and g^(k+1) may be represented as Equation(15):

$\begin{matrix}{\begin{matrix}{u_{xx}^{k + 1} = \left\{ \begin{matrix}{{g_{xx}^{k + 1} + v_{xx}^{k} - \frac{1}{\mu}},\ {{g_{xx}^{k + 1} + v_{xx}^{k}} \in \left( {\frac{1}{\mu},\infty} \right)}} \\{0,\ {{g_{xx}^{k + 1} + v_{xx}^{k}} \in \left( {{- \frac{1}{\mu}},\frac{1}{\mu}} \right)}} \\{{g_{xx}^{k + 1} + v_{xx}^{k} + \frac{1}{\mu}},\ {{g_{xx}^{k + 1} + v_{xx}^{k}} \in \left( {{- \infty},{- \frac{1}{\mu}}} \right)}}\end{matrix} \right.} \\{= {{shrink}\left( {{g_{xx}^{k + 1} + v_{xx}^{k}},\frac{1}{\mu}} \right)}}\end{matrix}{{u_{yy}^{k + 1} = {{{{shrink}\left( {{g_{yy}^{k + 1} + v_{yy}^{k}},\frac{1}{\mu}} \right)}u_{tt}^{k + 1}} = {{{{shrink}\left( {{{\lambda_{t} \cdot g_{tt}^{k + 1}} + v_{tt}^{k}},\frac{1}{\mu}} \right)}u_{xy}^{k + 1}} = {{{{shrink}\left( {{{2 \cdot g_{xy}^{k + 1}} + v_{xy}^{k}},\frac{1}{\mu}} \right)}u_{xt}^{k + 1}} = {{{{shrink}\left( {{{2{\sqrt{\lambda_{t}} \cdot g_{xt}^{k + 1}}} + v_{xt}^{k}},\frac{1}{\mu}} \right)}u_{yt}^{k + 1}} = {{{{shrink}\left( {{{2{\sqrt{\lambda_{t}} \cdot g_{yt}^{k + 1}}} + v_{\;^{yt}}^{k}},\frac{1}{\mu}} \right)}u^{k + 1}} = {{shrink}\left( {{{\lambda_{L1} \cdot g^{k + 1}} + v^{k}},\frac{1}{\mu}} \right)}}}}}}},}} & (15)\end{matrix}$

where u^(k+1) represents the intermediate variable u finally determinedin the last iteration. Using Equations. (13-15), the COP may be easilyresolved.

In some embodiments, for the preliminary image determined by theupsampling operation, the loss function (6) may be rewritten as follows:

$\begin{matrix}{{\underset{g}{\arg\mspace{14mu}\min}\left\{ {{\frac{\lambda}{2}{{f - b - {Dg}}}_{2}^{2}} + {R_{Hessian}(g)} + {\lambda_{L1}{g}_{1}}} \right\}},} & (16)\end{matrix}$

where D represents a downsampling matrix.

In 512, the processing device 140 (e.g., the deconvolution module 412)may generate a target image by performing, based on a second objectivefunction, a second iterative operation on the intermediate image.

In some embodiments, the intermediate image determined by the firstiterative operation may have low contrast in extrapolated spatialfrequency largely due to the convolution of the real object with the PSFof the image acquisition device 110 (e.g., 2D-SIM/SD-SIM) and theextension of bandwidth due to the sparsity and continuity constraints.The second iterative operation may be used to further boost the acquiredbandwidth of the intermediate image beyond that set by the imageacquisition device 110, e.g., eliminate the blur of the intermediateimage caused by the image acquisition device 110 itself.

In some embodiments, the processing device 140 may determine an initialestimated target image based on the intermediate image. The processingdevice 140 may update an estimated target image by performing, based onthe initial estimated target image, a plurality of iterations of thesecond object function. In each of the plurality of iterations, theprocessing device 140 may determine whether a third terminationcondition is satisfied. In response to determining that the thirdtermination condition is satisfied, the processing device 140 mayfinalize the target image and designate the estimated target image inthe iteration as the target image. In response to determining that thethird termination condition is not satisfied, the processing device 140may determine an updated estimated target image in the iteration. Theprocessing device 140 may perform a next iteration of the first objectfunction based on the updated estimated target image. The thirdcondition may be similar to the second condition, and not repeated here.

In some embodiments, the second objective function may be associatedwith a system matrix of the image acquisition device 110 and a seconddifference between the intermediate image and the target image. Thesecond difference may include a third difference between theintermediate image and the target image multiplied by the system matrix.For example, the second objective function may be expressed as a lossfunction as follows:

$\begin{matrix}{{\underset{x}{\arg\mspace{14mu}\min}\left\{ {{g - {Ax}}}_{2}^{2} \right\}},} & (17)\end{matrix}$

where g represents the intermediate image, A represents the systemmatrix, x represents the target image, Ax represents the target imagemultiplied by the system matrix, g-Ax represents the third difference,and ∥⋅∥₂ represents the L-2 norm.

In some embodiments, the second iterative operation may include aniterative deconvolution using a Landweber (LW) deconvolution algorithm,a Richardson Lucy (RL)-based deconvolution algorithm, or the like, orany combination thereof. For example, for the intermediate image with arelatively high SNR (e.g., the intermediate image in the 2D-SIM system)the processing device 140 may apply the Landweber (LW) deconvolution toprocess the intermediate image, in which a gradient descent algorithmwith Nesterov momentum acceleration may be used as follows:

$\begin{matrix}{{y^{({j + 1})} = {x^{({j + 1})} + {\beta_{k}\left( {x^{(j)} - x^{({j - 1})}} \right)}}}{x^{({j + 1})} = {y^{({j + 1})} + {A^{T}\left( {g^{N} - {Ay^{({j + 1})}}} \right)}}}{\gamma^{j} = {{\frac{1}{2}\left( {{4{\gamma^{2}}_{j - 1}} + {\gamma^{4}}_{j - 1}} \right)^{\frac{1}{2}}} - {\gamma^{2}}_{j - 1}}}{{\beta_{j} = {1 - {\gamma^{- 1}}_{j - 1}}},}} & (18)\end{matrix}$

where g^(N) represents the image after sparse reconstruction (i.e., theintermediate image) and x^(j+1) represents the target image after j+1iterations. In some embodiments, to attain reliable image fidelity, thesecond iterative operation may be terminated at an early stage toconstrain the target image. The numbers of deconvolution iterations(i.e., an iterative number of the second objective function) may belisted in a parameter table in a Software User Manual.

As another example, for the intermediate image with a relatively low SNR(e.g., the intermediate image in the SD-SIM system), the processingdevice 140 may apply the Richardson Lucy (RL)-based deconvolutionalgorithm to process the intermediate image, which is corrupted withexcessive noise. The RL-based deconvolution algorithm may be acceleratedexpressed as follows:

$\begin{matrix}{{y^{j + 1} = {x^{j} \cdot \left( {h^{T} \cdot \frac{g^{N}}{h \cdot x^{j}}} \right)}}{v^{j} = {x^{j + 1} - y^{j}}}{\alpha^{j + 1} = \frac{\sum{v^{j} \cdot v^{j - 1}}}{\sum{v^{j - 1} \cdot v^{j - 1}}}}{{x^{j + 1} = {y^{j + 1} + {\alpha^{j + 1} \cdot \left( {y^{j + 1} - y^{j}} \right)}}},}} & (19)\end{matrix}$

where g^(N) represents the image after sparse reconstruction (i.e., theintermediate image), x^(j+1) represents the target image after j+1iterations, and a denotes an adaptive acceleration factor representing acomputational iteration step length, which may be estimated directlyfrom experimental results. In some embodiments, taking advantage of thestable convergence performance, the required number of iterations in theaccelerated RL algorithm may be reduced by substantially 10 timescompared to the classical RL algorithm.

In some embodiments, as Equation (17) does not consider the effects ofPoisson distributed noise, the RL-deconvolution algorithm described inEquation (19) may not be the ideal model for sparse deconvolution;nonetheless, it may exhibit superior performance when processing realdata (e.g., the intermediate image). This may occur because the originaldata g, as seen in the first term of Equation (19), acts to constrainthe iterative deconvolution procedure, permitting relativelyhigh-fidelity restoration without introducing unexpected artifacts underlow SNR conditions.

In some embodiments, the target image may have an improved spatialresolution in comparison with a reference image generated based on athird objective function. The third objective function may include thefirst term and the second term of the first objective without the thirdterm of the first objective function. For example, the processing device140 may determine a second intermediate image by performing, based onthe third objective function, a fourth iterative operation (which issimilar to the first iterative operation) on the preliminary image. Theprocessing device 140 may generate the reference image by performing,based on the second objective function, a fifth iterative operation(which is similar to the second iterative operation) on the secondintermediate image. A first spatial resolution of the target image maybe improved by 50%-100% with respect to a second spatial resolution ofthe reference image.

It should be noted that the above description of the process 500 ismerely provided for purposes of illustration, and not intended to limitthe scope of the present disclosure. It should be understood that, afterunderstanding the principle of the operations, persons having ordinaryskills in the art may arbitrarily combine any operations, add or deleteany operations, or apply the principle of the operations to other imageprocessing process, without departing from the principle. In someembodiments, process 500 may include one or more additional operations.For example, an additional operation may be added after operation 512for displaying the target image. As another example, an additionaloperation may be added after operation 504 for pre-processing thepreliminary image. In some embodiments, one or more operations in theprocess 500 may be omitted. For example, operation 502, operation 506and/or operation 508 may be omitted. In some embodiments, two or moreoperations in the process 500 may be integrated into one operation. Forexample, operations 502 and 504 may be integrated into one operation. Asanother example, operations 506-510 may be integrated into oneoperation. In some embodiments, operations 510 and 512 may becollectively referred to as a sparse deconvolution operation (oralgorithm).

FIG. 6 is a flowchart illustrating an exemplary process for backgroundestimation according to some embodiments of the present disclosure. Insome embodiments, process 600 may be executed by the image processingsystem 100. For example, the process 600 may be implemented as a set ofinstructions (e.g., an application) stored in a storage device (e.g.,the storage device 150, the storage 220, and/or the storage unit 370).In some embodiments, the processing device 140 (e.g., the processor 210of the computing device 200, the CPU 340 of the mobile device 300,and/or one or more modules illustrated in FIG. 4) may execute the set ofinstructions and may accordingly be directed to perform the process 600.The operations of the illustrated process presented below are intendedto be illustrative. In some embodiments, the process may be accomplishedwith one or more additional operations not described, and/or without oneor more of the operations discussed. Additionally, the order in whichthe operations of the process as illustrated in FIG. 6 and describedbelow is not intended to be limiting. In some embodiments, one or moreoperations of the process 600 may be performed to achieve operation 506as described in connection with FIG. 5.

In 602, the processing device 140 (e.g., the background estimationmodule 406) may determine an input image based on a preliminary image.

In some embodiments, the processing device 140 may directly designatethe preliminary image as the input image. In some embodiments, theprocessing device 140 may pre-process the preliminary image anddesignate the pre-processed preliminary image as the input image. Forexample, the processing device 140 may determine the input image bydesignating pixel values in the preliminary image that is less than themean value of pixels in the preliminary image as zero. As anotherexample, the processing device 140 may determine the input image bycorrecting the preliminary image.

In 604, the processing device 140 (e.g., the background estimationmodule 406) may generate a decomposed image by performing a multilevelwavelet decomposition operation on the input image. As used herein, thedecomposed image may refer to an image in the frequency domain.

In some embodiments, the processing device 140 may perform themultilevel wavelet decomposition operation on the input image using oneor more wavelet filters as described in operation 506 in FIG. 5. Afterthe multilevel wavelet decomposition operation, the processing device140 may decompose the input image into a plurality of levels offrequency bands in the frequency domain. The processing device 140 mayextract the lowest frequency band from the plurality of levels offrequency bands. Further, the processing device 140 may generate thedecomposed image based on the lowest frequency band. For example, forthe 2D-SIM system, the processing device 140 may use a 2D Daubechies-6wavelet filter to decompose the preliminary image up to the 7^(th) levelin the frequency domain, i.e., the preliminary image may be decomposedinto 7 levels of frequency bands. The processing device 140 maydetermine the decomposed image based on the lowest frequency band amongthe 7 levels of frequency bands.

In 606, the processing device 140 (e.g., the background estimationmodule 406) may generate a transformed image by performing an inversewavelet transformation operation on the decomposed image. As usedherein, the transformed image may refer to an image in the spatialdomain. The inverse wavelet transformation operation may refer to anoperation that can transform information from the frequency domain tothe spatial domain, e.g., an inverse Fourier transformation operation.In some embodiments, operation 606 may be used to remove high-intensitypixels resulting from inaccurate estimation.

In some embodiments, to prevent accidental removal of minor usefulinformation (or signals), the inverse wavelet transformation operationmay be performed on the decomposed image (i.e., the lowest frequencyband) to generate the transformed image in the spatial domain. Forexample, the processing device 140 may perform the inverse wavelettransformation operation on the decomposed image to generate anintermediate transformed image. The processing device 140 may merge theintermediate image and half of the square root of the input image bykeeping the minimum values at each pixel to determine the transformedimage. That is, the processing device 140 may compare a value of eachpixel in the intermediate transformed image with half of the square rootof a value of a corresponding pixel in the input image. As used herein,a relative position of the corresponding pixel in the input image withrespect to the input image may be the same as a relative position of theeach pixel in the intermediate transformed image. If the value of aspecific pixel in the intermediate transformed image is greater thanhalf of the square root of the value of a corresponding pixel in theinput image, the processing device 140 may determine the transformedimage by replacing the value of the specific pixel in the intermediatetransformed image by the square root of the value of the correspondingpixel in the input image. If the value of the specific pixel in theintermediate transformed image is less than half of the square root ofthe value of the corresponding pixel in the input image, the processingdevice 140 may determine the transformed image by keeping the value ofthe specific pixel in the intermediate transformed image. Additionallyor alternatively, if the value of the specific pixel in the intermediatetransformed image is equal to half of the square root of the value ofthe corresponding pixel in the input image, the processing device 140may determine the transformed image by keeping the value of the specificpixel in the intermediate transformed image or replacing the value ofthe specific pixel in the intermediate transformed image by the squareroot of the value of the corresponding pixel in the input image.

In 608, the processing device 140 (e.g., the background estimationmodule 406) may generate an estimated image by performing a cut-offoperation on the transformed image. In some embodiments, the estimatedimage may also be referred to as a low peak background estimation.

After the cut-off operation, amplitude values of the transformed imagemay be cut off in a certain percentage in the frequency domain. That is,amplitude values of the estimated image in the frequency domain may bethe certain percentage of the amplitude values of the transformed image.For example, the processing device 140 may perform a 50% cut-offoperation on the transformed image. The amplitude values of theestimated image may be 50% of the amplitude values of the transformedimage. As another example, the processing device 140 may perform a 40%cut-off operation on the transformed image. The amplitude values of theestimated image may be 40% of the amplitude values of the transformedimage.

In 610, the processing device 140 (e.g., the background estimationmodule 406) may determine whether a termination condition (i.e., thefirst termination condition in FIG. 5) is satisfied. In response todetermining that the first termination condition is satisfied, theprocessing device 140 may proceed to operation 612. In response todetermining that the first termination condition is not satisfied, theprocessing device 140 may designate the estimated image determined inoperation 608 as a new input image for iteratively performing operations604-608.

In some embodiments, the first termination condition may be associatedwith an iterative number (count) of operations 604-608. For example, thefirst termination condition may be satisfied if the iterative number isequal to a threshold count (e.g., 3). In some embodiments, the firsttermination condition may be associated with a difference between twoestimated images determined in two consecutive iterations. For example,the first termination condition may be satisfied if the differencebetween two estimated images determined in two consecutive iterations isless than a threshold. In some embodiments, the first terminationcondition may be associated with the iterative time of total iterations.For example, the first termination condition may be satisfied if theiterative time of all iterations is less than a threshold.

In 612, the processing device 140 (e.g., the background estimationmodule 406) may determine the estimated image as an estimated backgroundimage of the preliminary image. Further, the processing device 140 maygenerate a residual image by extracting the estimated background imagefrom the preliminary image for sparse deconvolution as described inoperations 508-512.

It should be noted that the above description of the process 600 ismerely provided for purposes of illustration, and not intended to limitthe scope of the present disclosure. It should be understood that, afterunderstanding the principle of the operations, persons having ordinaryskills in the art may arbitrarily combine any operations, add or deleteany operations, or apply the principle of the operations to other imageprocessing process, without departing from the principle. In someembodiments, one or more operations may be added or omitted in theprocess 600. For example, an additional operation may be added forstoring the estimated background image.

In some embodiments, fluorescence microscopy may achieve high resolutionand contrast by labeling designated structures such as actin,microtubules, and vesicles, which may often be sparse in the spatialdomain in terms of the existence of a fraction of nonzero value.However, equivalent to the filtering of high-frequency information bythe OTF of the microscope, the final captured fluorescence images (e.g.,the raw images, or SR images) may be blurred by the point spreadfunction (PSF) and exhibit an increase in nonzero values. In thisregard, enhanced spatial resolution may mean accepting sparsityincreases in SR images. However, previous attempts to recover SRinformation using prior knowledge of sparsity may suffer from typicallyunstable one-step reconstruction, which depends logarithmically on theimage signal-to-noise ratio (SNR).

In the present disclosure, a three-step procedure that both boostsresolution and maintains the fidelity of the SR information is providedas outlined in FIG. 7 and FIG. 12. First, the raw images may beinversely filtered with the conventional Wiener algorithm to maximallyboost the SNR of the SR components within the assembled OTF asdetermined by the image acquisition device 110 (e.g., as described inoperation 504 in FIG. 5) and removed background fluorescence originatingfrom out-of-focus planes (e.g., as described in operation 506 in FIG. 5or operations in FIG. 6). For images (e.g., the residual image) that didnot meet the Nyquist sampling criteria, a manual upsampling step (e.g.,as described in operation 508) may be added.

Second, to enhance reconstruction fidelity, the Hessian penalty on thecontinuity along the xy-t axes may be combined with the sparsity term(the l₁ norm of the sum of the absolute values of each element) toiteratively approximate the SR information that extends beyond the OTF(e.g., as described in operation 510 in FIG. 5). The sparsereconstruction/deconvolution may use constrained optimization togenerate an optimized image g (i.e., the intermediate image) fromWiener-filtered original SIM image f (i.e., the preliminary image). Theimage (i.e., the preliminary image) may be first corrected forbackground b (detailed procedures may be presented in operation 506 inFIG. 5 or operations in FIG. 6). Then, g may be generated by iterativelyminimizing the following loss function:

$\begin{matrix}{{\underset{g}{argmin}\left\{ {{\frac{\lambda}{2}{{f - b - g}}_{2}^{2}} + {R_{Hessian}(g)} + {\lambda_{L1}{g}_{1}}} \right\}},} & (6)\end{matrix}$

where the first term represents the conventional squared deviation ofthe reconstructed image (i.e., the intermediate image) from thebackground-corrected filtered data (i.e., the residual image), thesecond term represents the continuity constraint as a penalty for highsecond spatial derivatives (Hessian curvature) of the resulting image(i.e., the intermediate image), and the third term represents thesparsity constraint as a penalty for excessive nonzero values. ∥⋅∥₁ and∥⋅∥₂ respectively represent the

and

norms, and λ and λ_(L1) respectively represent the weighting factorsthat balance the fidelity and sparsity of the images. The loss functionmay form a convex optimization problem solvable using the split Bregmanalgorithm (detailed equations and procedures are shown in operation 510in FIG. 5). While the sparsity term (the summation of the absolutevalues from all pixels within the image stack) permits someextrapolations of bandwidth limited by the image acquisition device 110(FIGS. 13, 14, 15, 16, and 17), the fidelity term (the Hessian penaltyon the continuity along the xy-t axes) may ensure a reconstruction witha robust SNR against artifacts (FIGS. 14 and 17).

Finally, another iterative deconvolution step (e.g., as described inoperation 512 in FIG. 5) may be applied based on maximum a posterioriestimation (MAP) to further enhance the contrast of the high-frequencycomponents that extends beyond the image acquisition device110-determined OTF (detailed procedures being presented in operation 512in FIG. 5).

The present disclosure is further described according to the followingexamples, which should not be construed as limiting the scope of thepresent disclosure.

EXAMPLES Materials and Methods

Cell Maintenance and Preparation.

INS-1 cells were cultured in RPMI 1640 medium (GIBCO, 11835-030)supplemented with 10% FBS (GIBCO), 1% 100 mM sodium pyruvate solution,and 0.1% 55 mM 2-mercaptoethanol (GIBCO, 21985023) in an incubator at37° C. with 5% CO₂ until ˜75% confluency was reached. COS-7 cells andHeLa cells were cultured in high-glucose DMEM (GIBCO, 21063029)supplemented with 10% fetal bovine serum (FBS) (GIBCO) and 1% 100 mMsodium pyruvate solution (Sigma-Aldrich, S8636) in an incubator at 37°C. with 5% CO₂ until ˜75% confluency was reached. For the 2D-SIM imagingexperiments, cells were seeded onto coverslips (H-LAF 10 L glass,reflection index: 1.788, diameter: 26 mm, thickness: 0.15 mm,customized). For the SD-SIM imaging experiments, 25-mm coverslips(Fisherbrand, 12-545-102) were coated with 0.01% Poly-L-lysine solution(SIGMA) for ˜24 hours before seeding transfected cells.

For the 2D-SIM experiments, to label LEs/LYs, COS-7 cells were incubatedwith 1× LysoView™ 488 (Biotium, 70067-T) in complete cell culture mediumat 37° C. for 15-30 min and protected from light, without being washedand imaged. To label LEs/Lys in the SD-SIM experiments, COS-7 cells wereincubated in 50 nM LysoTracker Deep Red (Thermo Fisher Scientific,L12492) for 45 min and washed them 3 times in PBS before imaging. Tolabel LDs, COS-7 cells were incubated with 1× LipidSpot™ 488 (Biotium,70065-T) in complete cell culture medium at 37° C. for 30 min andprotected from light before being washed and imaged. To labelmitochondria, COS-7 cells were incubated with 250 nM MitoTracker™ GreenFM (Thermo Fisher Scientific, M7514) and 250 nM MitoTracker® Deep Red FM(Thermo Fisher Scientific, M22426) in HBSS containing Ca²⁺ and Mg²⁺ butno phenol red (Thermo Fisher Scientific, 14025076) at 37° C. for 15 minbefore being washed three times before imaging. To perform nuclearstaining, COS-7 cells were incubated with 10 μg/ml Hoechst (ThermoFisher Scientific, H1399) in PBS for ˜5 mins and washed 3 times in PBS.

To label cells with genetic indicators, Cos-7 cells were transfectedwithcaveolin-EGFP/LifeAct-EGFP/LAMP1-EGFP/LAMP1-mChery/Tom20-mScarlet/Tom20-mCherry/Sec61β-Cherry/Sec61β-EGFP/clathrin-EGFP/clathrin-DsRed.HeLa cells were transfected with Tubulin-EGFP/Pex11a-BFP/LAMP1-mChery.INS-1 cells were transfected with Vamp2-pHluorin. The transfections wereexecuted using Lipofectamine™ 2000 (Thermo Fisher Scientific, 11668019)according to the manufacturer's instructions. After transfection, cellswere plated in precoated coverslips. Live cells were imaged in completecell culture medium containing no phenol red in a 37° C. live cellimaging system. For the experiments conducted with INS-1 cells, thecells were stimulated with a solution containing 70 mM KCl and 20 mMglucose and vesicle fusion events were observed under a TIRF-SIMmicroscope.

The Interference-Based SIM Setup.

The schematic illustration of the system is based on a commercialinverted fluorescence microscope (IX83, Olympus) equipped with a TIRFobjective (Apo N 100×/1.7 HI Oil, Olympus), a wide-field objective(100×/1.45 Oil, ApoN, Olympus) and a multiband dichroic mirror (DM,ZT405/488/561/640-phase R; Chroma). In short, laser light withwavelengths of 488 nm (Sapphire 488LP-200) and 561 nm (Sapphire561LP-200, Coherent) and acoustic optical tunable filters (AOTF, AAOpto-Electronic, France) were used to combine, switch, and adjust theillumination power of the lasers. A collimating lens (focal length: 10mm, Lightpath) was used to couple the lasers to apolarization-maintaining single-mode fiber (QPMJ-3AF3S, Oz Optics). Theoutput lasers were then collimated by an objective lens (CFI PlanApochromat Lambda 2× NA 0.10, Nikon) and diffracted by the pure phasegrating that consisted of a polarizing beam splitter (PBS), a half waveplate and the SLM (3DM-SXGA, ForthDD). The diffraction beams were thenfocused by another achromatic lens (AC508-250, Thorlabs) onto theintermediate pupil plane, where a carefully designed stop mask wasplaced to block the zero-order beam and other stray light and to permitpassage of ±1 ordered beam pairs only. To maximally modulate theillumination pattern while eliminating the switching time betweendifferent excitation polarizations, a home-made polarization rotator wasplaced after the stop mask (18). Next, the light passed through anotherlens (AC254-125, Thorlabs) and a tube lens (ITL200, Thorlabs) to focuson the back focal plane of the objective lens, which interfered with theimage plane after passing through the objective lens. Emittedfluorescence collected by the same objective passed through a dichroicmirror (DM), an emission filter and another tube lens. Finally, theemitted fluorescence was split by an image splitter (W-VIEW GEMINI,Hamamatsu, Japan) before being captured by an sCMOS (Flash 4.0 V3,Hamamatsu, Japan).

The SD-SIM Setup.

The SD-SIM is a commercial system based on an inverted fluorescencemicroscope (IX81, Olympus) equipped with a 100×, 1.3 NA) oil objectiveand a scanning confocal system (CSU-X1, Yokogawa). Four laser beams of405 nm, 488 nm, 561 nm, and 647 nm were combined with the SD-SIM. TheLive-SR module (GATACA systems, France) was equipped with the SD-SIM.The images were captured either by an sCMOS (C14440-20UP, Hamamatsu,Japan) or an EMCCD (iXon3 897, Andor) as mentioned in FIG. 10 or FIG.11.

FRC Resolution Map.

To analyze the imaging capability of SR microscopes, FRC resolution maybe implemented to describe the effective resolution. To visualize theFRC resolution more clearly, the blockwise FRC resolution mapping may beapplied to evaluate the (Sparse) SIM and (Sparse) SD-SIM images. Morespecifically, each image may be divided into independent blocks, and thelocal FRC value may be calculated individually. If the FRC estimationsin a block are sufficiently correlated, this region may be color codedin the FRC resolution map. Otherwise, the region may be color codedaccording to its neighbor interpolation. It should be noted that beforecalculating the FRC resolution map of SD-SIM raw images in FIGS. 10A,10H and 11A, a wavelet denoise algorithm may be applied in advance toimages to avoid the ultralow SNR of SD-SIM raw images influencing theFRC calculation.

FWHM and Rayleigh Criterion Analysis.

All the datasets extracted from the full width at half-maxima (FWHM) andtwo peak fitting may be processed using the multiple peaks fit inOriginPro with Gaussian functions. To avoid the influence of large beadson the bead resolution estimation, a bead-size correction factor may beincluded in the FWHM resolution calibration for (Sparse) SD-SIM (FIG.18). Therefore, the exact resolution value may be calculated using thefollowing equation:

FWHM_(r)=√{square root over (FWHM_(m) ² −d ²)},   (20)

where FWHM_(m) denotes the FWHM estimated directly from a single pointin the image, FWHM_(r) denotes the corrected resolution of (Sparse)SD-SIM, and d represents the bead diameter, which is 100 nm in FIG. 18.

Mesh Pore Diameters of Actin Networks.

The mesh pore diameters (FIG. 8H) of actin networks may be analyzed.Considering the low SNR condition and high background in 2D-SIM andSD-SIM images, the pre-transform may be replaced with the pre-filter.Specifically, a Gaussian filter and an unsharp mask filter may be usedsuccessively to denoise the raw image and remove its background. Thepores may be extracted by Meyer watershed segmentation. Then, the poresizes may be calculated by the number of enclosed pixels from invertedbinary images multiplied by the physical pixel size. In someembodiments, the Sparse-SIM and Sparse SD-SIM images (i.e., target SIMand SD-SIM images determined by sparse reconstruction) may be segmenteddirectly using hard thresholds due to their extremely clear backgrounds.

CCP Diameters.

Two main procedures may be involved before calculating the CCP diametersin FIG. 10A. First, the image in FIG. 10A may be segmented with thelocally adaptive threshold to segment the ring structures in the image.Second, the resulting binary images may be processed by the Meyerwatershed after distance transform to separate touching CCPs, and thegoal may be to generate a boundary as far as possible from the center ofthe overlapping CCPs. Subsequently, the CCP diameters may be calculatedin the same manner as the mesh pore diameters.

Image Rendering and Processing.

The color map SQURREL-FRC associated with ImageJ may be used to presentthe FRC map in FIG. 8E. The color map Morgenstemning may be applied toexhibit the lysosome in FIG. 10H and the Fourier transform results inFIG. 11D. A 16-color projection may be used to show the depth in FIG.10K, and jet projection may be used to show the depth in FIG. 11K andits time series in FIG. 10G. The volume in FIG. 11H may be rendered byClearVolume. All data processing may be achieved using MATLAB andImageJ. All the Figures may be prepared with MATLAB, ImageJ, MicrosoftVisio, and OriginPro, and videos may all be produced with a self-writtenMATLAB library, which is available athttps://github.com/WeisongZhao/img2vid.

Adaptive Filter for SD-SIM.

Confocal type images may often exhibit isolated pixels (1×1˜5×5) withextremely bright values caused by voltage instability or dead/hot camerapixels. The magnitudes of these pixels may be approximately 5 to 100times higher than the normal intensity amplitudes of the biostructure.These isolated pixels may be ill-suited for the sparse reconstruction.An adaptive median filter may be created to remove these improperpixels. More specifically, instead of the normal median filter, whichreplaces each pixel with the median of the neighboring pixels in thewindow, a threshold for the developed adaptive median filter may be set.If the pixel intensity is larger than thresholdxmedian in the window,the pixel may be replaced by the median; otherwise, the window may moveto the next pixel. By using this algorithm, the isolated pixels may befiltered without blurring the images. The related algorithm may bewritten as an ImageJ plug-in, and may be found athttps://github.com/WeisongZhao/AdaptiveMedian.imagej.

Software and GPU Acceleration.

Sparse-SIM may be implemented in MATLAB using a CPU (Intel i9-7900X, 3.3GHz, 10 cores, 20 threads, and 128 GB memory), the NVIDIA CUDA fastFourier transform library (cuFFT), and a GPU (TITAN RTX, 4608 CUDA coresand 24 GB memory). The FFT may be utilized to accelerate thehigh-content and multidimensional matrix convolution operation inSparse-SIM. Under the premise that the GPU memory is sufficient for theinput data, the effect of acceleration may become more obvious as thesize of the data increases. In most circumstances, a 60-fold improvementmay be achieved with GPU acceleration compared to CPU processing. Forexample, the image dataset of 1536×1536×20 in FIG. 8A requiredsubstantially 90 s to process with the GPU, far less than 46 mins neededfor processing with the CPU. The version of Sparse-SIM software used inthe present disclosure (accompanied with the example data, and usermanual) may be available as Supplementary Software. The updated versionof Sparse-SIM software may be found athttps://github.com/WeisongZhao/Sparse-SIM.

Example 1 Validation of Resolution Extension By Sparse Deconvolution

First, the extension of spatial resolution may be tested by thedeconvolution algorithm on synthetic filaments with known ground truth.As shown in FIG. 13, two filaments substantially 110 nm apart may beseparated after Wiener inverse filtering, while filaments closer thansubstantially 100 nm may hardly be resolved. Iterative imagereconstruction with the sparsity prior may create only a smalldifference in fluorescence of the middle part between the two filaments,while the deconvolution resulted in the clear separation of twofilaments up to substantially 81 nm apart (FIGS. 13A and 13B). However,the contrast for two filaments substantially 65 nm apart may be low(possibly limited by the pixel size of 32 nm), which may be furtherimproved after the pixel upsampling procedure (FIGS. 13C and 13D). Next,the functionality of different steps in the algorithm may be tested onsynthetic filament structures corrupted with different levels of noise.Deconvolution without the addition of the sparsity prior may be unableto retrieve the high-frequency information trimmed off by the SIM OTF,while deconvolution without the addition of the continuity prior maylead to reconstruction artifacts that manifested particularly in rawimages with low SNR (FIG. 14). However, the combination of thecontinuity and sparsity priors may enable robust and high-fidelityextrapolation of the high-frequency information inaccessible to SIM,even under situations with excessive noise (FIG. 14, Tables S1 and S2).

Finally, the ability of sparse deconvolution may be tested to increasethe resolution of wide-field microscopy and the fidelity of newlyidentified biological structures. By convolving the image obtained bythe 1.4 NA objective with a synthetic PSF from a 0.8 NA objective, ablurred version of actin filaments (FIGS. 15A and 15B) may be created.Interestingly, two blurry opposing actin filaments under the low NAobjective may become clearly separable after sparse deconvolutioncoupled with an extended Fourier frequency domain (FIGS. 15C and 15D).Likewise, instead of simply shrinking the diameter of actin filaments,two neighboring filaments that are substantially 120 nm apart (confirmedby 2D-SIM) may be resolved by the sparse deconvolution of wide fieldimages (FIG. 15E). In addition, a CCP of ˜135 nm in diameter under theTIRF-SIM may manifest as a blurred punctum in wide-field images that mayhave undergone conventional deconvolution or been deconvolved with theHessian continuity prior. Only after sparse deconvolution, thering-shaped structure may emerge (FIG. 15F). All these data may confirmthe validity of the extension in spatial resolution, as alsodemonstrated by the extended OTF obtained under different conditions(FIG. 16).

Example 2 Sparse-SIM Achieves 60 nm Resolution Within MillisecondExposures in Live Cells

Actin may be a major cellular cytoskeletal structural building blockthat constitutes the backbone of the cell membrane and defines the shapeand general organization of the cytoplasm. It may be known thatconcentrated actin may form a gel-like network under the cell cortex andbe dynamically regulated during such processes as cell division,migration, and polarization formation. Although actin meshes with poresize, diameters of approximately 50˜200 nm may be detected with electronmicroscopy; however, the dynamic reshaping of these pores may hardly becaptured by any live cell SR imaging algorithm due to the lack ofsufficient spatiotemporal resolution. Under wide-field illumination, twoactin filaments substantially 66 nm apart, which may beindistinguishable with either 2D-SIM or Hessian SIM, may become clearlyresolved after deconvolution via the entire algorithm (as shown in FIGS.8A and 8B) but not when only a subset of the steps is applied (FIG. 17).Better separation of dense actin may mesh results from both the enhancedcontrast (as shown in FIG. 8D) and the increased spatial resolution, asshown by both full-width-at-half-maximum (FWHM) analysis of actinfilaments and Fourier Ring Correlation (FRC) mapping analysis of SRimages (as shown in FIGS. 8E and 8F). During time-lapse SR imaging ofactin filament dynamics, the resolution of Sparse-SIM may remain stable,as demonstrated by the relatively consistent values obtained by theFRC-based method (as shown in FIG. 8G). The increase in resolution maybe well-demonstrated by the more frequent observations of small poreswithin the actin mesh. The mean diameter of pores within the corticalactin mesh may be substantially 160 nm from the Sparse-SIM data (asshown in FIG. 8I). An increase in the spatial resolution may also helpresolve the ring-shaped caveolae (diameter substantially 60 nm) underTIRF illumination in live COS-7 cells (as shown in FIGS. 8J-8L).Previously, this required nonlinear SIM may come at the expense ofadditional reconstruction artifacts and limited imaging durations. Forfluorescent vesicles such as lysosomes and lipid droplets within thecytosol under wide-field illumination, the sparse deconvolution mayeliminate both the background and the reconstruction artifacts due tothe low SNR. These data may show that the contrast and resolution may bemaintained deep in the cytosol, and the SR images may be sharper thanthose obtained with conventional SIM or Hessian SIMs (as shown in FIGS.8M and 8N).

Finally, from raw images captured at a 1,692 Hz frame rate and rollingreconstruction with a temporal resolvability of 564 Hz, sparsedeconvolution may be used to provide a clearer view of vesicle fusionevents labeled by VAMP2-pHluorin in INS-1 cells. Sparse-SIM may alsodistinguish open fusion pores as small as substantially 60 nm indiameter (as shown in FIGS. 8O and 8P)—much smaller than thosedetectable by conventional or Hessian SIM (which is on averagesubstantially 130 nm in diameter, as shown in FIG. 8Q). Likewise,compared with fusion pores detected by the conventional SIM, Sparse-SIMmay detect pores occurred much earlier (substantially 40 ms versussubstantially 69 ms after the initial opening of the fusion pore) andthe images may last for a much longer time during the whole fusionprocess (substantially 182 ms versus substantially 82 ms, as shown inFIG. 8Q). These data may support a spatiotemporal resolution ofSparse-SIM of substantially 60 nm and 564 Hz, which may have notcurrently achievable using other existing SR algorithms.

Example 3 Relative Movements Between Suborganellar Structures ObservedBy Dual-Color Sparse-SIM

As a purely mathematical SR technique, Sparse-SIM may be readily used ina dual-color imaging paradigm to improve the resolution at bothwavelengths. For example, in cells with both outer and the innermitochondrial membranes (OMM and IMM) labeled with Tom20-mCherry andMito-Tracker Green, Sparse-SIM may show sharper cristae structures thanthe conventional 2D-SIM (as shown in FIG. 9A). In addition, the improvedresolution may also permit the detection of small differences betweenthe OMM and the IMM in live mitochondria. At 60˜70 nm resolution,Sparse-SIM may readily detect two types of OMM:IMM configurations, e.g.,a long crista extended from one side to substantially 142 nm away fromthe other side of the OMM, and a short crista extended onlysubstantially 44 nm towards the other side of the OMM (as shown in FIGS.9B-9D). Live cell SR imaging may also reveal rare events, such as theIMM extension not being enclosed within the Tom20-labeled structures ina few frames (as shown in FIG. 9E). This result may be explained by thenonhomogenous distribution of Tom20 protein on the OMM.

Dual-color Sparse-SIM imaging may also resolve more details regardingorganelle contacts, including those formed between the mitochondria andER. With the better identification of mitochondrial cristae, it may befound that ER tubules (labeled by Sec61β-mCherry) may randomly contactthe mitochondria with equal probability at both the cristae regions andthe regions between cristae (as shown in FIG. 9F). In addition, thecontact between one ER tubule and the top of a mitochondrion may notonly correlate with the directional movement of the latter but also thesynergistically rearranged orientations of cristae thereafter (as shownin FIG. 9G). Whether such cristae remodeling is causally linked withER-mitochondrial contacts may remain to be explored in the future, whilethe ability to visualize these subtle structural changes may lead to newinsights into the functional regulation of mitochondria.

Example 4 Sparse Deconvolution May Also Be Used to Increase theThree-Dimensional Resolution of Spinning-Disc Confocal-Based SIM(SD-SIM)

Because the continuity and sparseness assumptions are general featuresof SR fluorescence microscopy, the algorithm may be tested on SR imagedata obtained by SD-SIM, the pinhole-based SIM known by commercial namessuch as Airyscan, Live-SR, SpinSR10, Re-Scan Confocal Microscope, etc.Despite a spatial resolution increase of 1.5-1.7-fold over thediffraction limit, SD-SIM may suffer from a low photon budget and poorsignal-to-noise ratio (SNR) in live cells because the excitation andemission efficiencies are compromised by the microlens and pinholearrays. By using the sparse deconvolution on 3D image stacks offluorescent beads (100 nm) acquired by SD-SIM and correcting for thesize effects of fluorescent beads, a lateral resolution of substantially82 nm and an axial resolution of substantially 266 nm (as shown in FIG.18) may be obtained, a nearly twofold increase of spatial resolution inall three axes compared to the SD-SIM.

In live COS7-cells labeled with Clathrin-EGFP, Sparse SD-SIM may enablea blurred fluorescent punctum to be resolved as a ring-shaped structurewith a diameter of 89 nm (as shown in FIGS. 10A and 10B), which mayagree with the resolution measured by the FRC map algorithm (as shown inFIG. 10C). The average diameter of all the ring-shaped CCPs was ˜154 nm(as shown in FIG. 10D), the same as previously measured with high NATIRF-SIM. Events such as the disappearance of a ring-shaped CCP and thedisintegration of another CCP into two smaller rings nearby may be seen(as shown in FIGS. 10E and 10F). These rich CCP dynamics may be betterinterpreted by simultaneously labeling and imaging other molecules, suchas actin filaments (labeled by LifeAct-EGFP). Because photon budgetallowed by the Sparse SD-SIM could be as small as was substantially 0.9W/cm² (Table S3), both actin filaments and CCPs within a large field ofview of 44 μm×44 μm may be monitored for more than 15 min at a timeinterval of 5 s (as shown in FIGS. 10G and 19). Under these conditions,many relative movements between CCPs and filaments may be seen. Forexample, the de novo appearance and the stable docking of a CCP at theintersection of two actin filaments may be observed, followed by itsdisappearance from the focal plane as the neighboring filaments mayclose up and join together (as shown in FIG. 19D), which may beconsistent with the role of actin in the endocytosis of CCPs. Similarly,dual-color Sparse SD-SIM may also reveal dynamic interplays between ERtubules and lysosomes, such as the hitchhiking behavior (as shown inFIG. 20).

Less likely to be affected by chromatic aberration, Sparse SD-SIM mayeasily be adapted to four-color SR imaging, allowing the dynamics oflysosomes, mitochondria, microtubules, and nuclei to be monitored inlive cells (as shown in FIGS. 8H and 8I) at FRC resolutions as small as79˜90 nm (as shown in FIG. 4J). Benefiting from the absence ofout-of-focus fluorescence and the improved axial resolution, SparseSD-SIM may allow similar structures to be seen at similar contrastlevels throughout the cells, such as the mitochondrial outer membranestructures with FWHMs at substantially 280 nm axially (as shown in FIG.8L) maintained in the live cell that is substantially 7 μm thick (asshown in FIG. 8K).

Example 5 Upsampling Helped Sparse Deconvolution Increase ResolutionUnder Insufficient Nyquist Sampling Regimes

Equipped with superior quantum efficiency and electron multiplyingcapability, EMCCD cameras may be often used in SD-SIM systems, but maylimit the system resolution due to their large pixel sizes. For example,when imaging ER tubules under the SD-SIM deconvolve with the sparsealgorithm, FRC mapping analysis may yield a resolution of substantially195 nm, mostly due to the Nyquist limit of the pixel size ofsubstantially 94 nm (as shown in FIGS. 11A-11B). By artificiallyupsampling the data before deconvolution, previously blurred ring-shapedER tubules may become distinguishable (as shown in FIG. 115C). The OTFof the microscope may be expanded after upsampling and deconvolution (asshown in FIG. 11D), which may agree with the resolution increase tosubstantially 102 nm according to the FRC measurements (as shown in FIG.11B).

Upsampling before sparse deconvolution (Sparsity ×2) may also beextended to multicolor Sparse SD-SIM imaging to follow the dynamicsamong different organelles. Using HeLa cells coexpressing LAMP1-mCherry,Pex11a-BFP, and Tubulin-EGFP, the structures and dynamics of lysosomes,peroxisomes, and microtubules (as shown in FIG. 6E) may follow. Manyperoxisomes may encounter lysosomes on microtubules, demonstrated by alysosome moving along the microtubules and colliding with twoperoxisomes stably docked closely to the intersection of two tubulinfilaments (as shown in FIG. 11F), or the co-migration of a lysosome anda peroxisome along a microtubule for some time before separation anddeparture (as shown in FIG. 11G). These contacts may mediate lipid andcholesterol transport, as reported previously.

Finally, it may show that the XZ resolution of SD-SIM may also increaseby the Sparsity×2 procedure. In COS-7 cells labeled with Hoechst,MitoTracker Deep Red and Tubulin-EGFP, nuclei, mitochondria andmicrotubules in a three-dimensional volume spanning substantially 6 μmin the axial axis of a live cell (as shown in FIG. 11H) may be observed.Interestingly, the axial FWHM of a microtubule filament may decreasefrom substantially 465 nm in the raw dataset to substantially 228 nmafter Sparsity×2 processing, again indicative of a twofold increase inthe Z-axis (as shown in FIG. 11I). From the volumetric reconstruction,it may be apparent that the continuous, convex nuclear structure maybend inward and become concave at regions in some axial planes that wereintruded by extensive microtubule filaments and mitochondria (as shownin FIG. 11J). Such reciprocal changes may suggest that the tubulinnetwork may affect nucleus assembly and morphology. Overall, thecombination of sparse deconvolution with SD-SIM enables multicolor,volumetric SR imaging with minimal phototoxicity and aberrations; theseimages may provide more dynamic insights into cellular processes.

FIG. 7 illustrates exemplary results indicating a flowchart of thesparse deconvolution algorithm according to some embodiments of thepresent disclosure. Raw images from 2D-SIM or SD-SIM microscopes may beWiener inverse filtered to generate SR images, which may be backgroundsubtracted, upsampled, and reconstructed with the sparsity and thecontinuity priori information along the xy-t(z) axials before the finaliterative deconvolution. Scale bar: 1 μm.

FIGS. 8A-8Q illustrate exemplary results indicating sparse-SIM mayachieve substantially 60 nm and millisecond spatiotemporal resolution inlive cells according to some embodiments of the present disclosure. FIG.8A shows a representative COS-7 cell labeled with LifeAct-EGFP observedunder 2D-SIM (bottom), Hessian-SIM (top) and Sparse-SIM (middle). FIG.8B shows enlarged regions enclosed by box 810 in FIG. 8A (left: 2D-SIM;middle: Hessian-SIM; right: Sparse-SIM). FIG. 8C shows the correspondingprofiles of lines 830-850 in FIG. 8B. FIG. 8D shows average contrasts oftwo peaks with different distances under 2D-SIM and Sparse-SIM. Thecontrast may be calculated following the equation (I max−I min)/(I max+Imin). FIG. 8E shows FRC maps of actin filaments under 2D-SIM andSparse-SIM. FIG. 8F shows resolutions of 2D-SIM and Sparse-SIM measuredas the FWHMs of the narrowest actin filaments and by the FRC techniques.FIG. 8G shows time-dependent changes in minimal FRC resolutions of2D-SIM images (black hollow circle), Hessian-SIM images (gray hollowcircle), and Sparse-SIM images (gray hollow square) against thetheoretical limit of 2D-SIM (black hollow triangle). FIG. 8H shows themagnified view of actin filaments in box 810 from FIG. 8A appears on theleft, while the segmented filaments are overlaid with relatively thinlines on the right. FIG. 8I shows cumulative distributions of pore sizeswithin the actin meshes in FIG. 8H measured from 2D-SIM (black) andSparse-SIM (gray) images. FIG. 8J shows representative example of aCOS-7 cell labeled with caveolin-EGFP under TIRF-SIM (left,substantially 98 nm FRC mean resolution) and Sparse-SIM (right,substantially 58 nm FRC mean resolution). As shown in FIG. 8K, from topto bottom may be magnified views of the white box in FIG. 8Jreconstructed by TIRF-SIM, Sparse-SIM (pixel size of 32 nm), andSparse-SIM with 2×upsampling (pixel size of 16 nm). FIG. 8L showsfluorescence profiles of the lines 860-880 in FIG. 8K. In live COS-7cells, as shown in FIG. 8M, cytosolic lysosomes may be labeled withLAMP1-EGFP (left, yellow) or LysoView (middle, cyan), while lipiddroplets may be labeled with LipidSpot (right, magenta). Top: Comparisonof the same structures observed under 2D-SIM (left) and Sparse-SIM(right). Bottom: Magnified views reconstructed by 2D-SIM (left),Hessian-SIM (middle), and Sparse-SIM (right). FIG. 8N shows averagediameters of caveolin-EGFP-, LAMP1-EGFP-, LysoView- andLipidSpot-labeled structures detected by Sparse-SIM. FIG. 8O showsrepresentative montages of vesicle fusion events observed under TIRF-SIM(top) and Sparse-SIM (bottom) configurations. The INS-1 cells may belabeled with VAMP2-pHluorin, and SR imaging may be conducted at a framerate of 564 Hz. FIG. 8P shows kymographs from lines in the TIRF-SIM(upper) and Sparse-SIM (lower) images shown in FIG. 8O. FIG. 8Q showsaverage pore diameters (left), pore opening times (middle), and poreduration times (right) measured under the TIRF-SIM (red) and Sparse-SIM(cyan) configurations. Center line, medians; limits, 75% and 25%;whiskers, maximum and minimum; error bars, s.e.m.; Cumul. Freq.,cumulative frequency; scale bars: (FIGS. 8A, 8E, and 8M top) 5 μm;(FIGS. 8B and 8J) 500 nm; (FIGS. 8H and 8M bottom) 1 μm; (FIGS. 8K and8O) 100 nm; (FIG. 8P) 500 ms.

FIGS. 9A-9G illustrate exemplary results indicating intricate dynamicswithin the mitochondria and between ER-mitochondria visualized bydual-color Sparse-SIM according to some embodiments of the presentdisclosure. FIG. 9A shows a representative example of a COS-7 cell inwhich both the outer (magenta) and the inner (cyan) mitochondrialmembrane (OMM and IMM) were labeled with Tom20-mScarlet and MitoTrackerGreen and observed under 2D-SIM (left) and Sparse-SIM (right). FIG. 9Bshows magnified views of the IMM (left) and the OMM (right) from box 910in FIG. 9A. FIG. 9C shows fluorescence intensity profiles ofTom20-mScarlet (magenta) and MitoTracker Green (cyan) along thecontinuous and dashed lines in FIG. 9B, which correspond to the twomitochondrial configurations shown in the right-hand schematic diagram(indicated by the black arrows). FIG. 9D shows average minimalresolutions of IMM (cyan) and OMM (magenta) in FIG. 9A observed under2D-SIM and Sparse-SIM. Twenty images along the t-axis may be used forthe FRC analysis. As shown in FIG. 9E, the white dashed box 920 in FIG.9A may be enlarged and shown at three time points, revealing thenonhomogenous distribution of Tom20 (right) relative to MitoTrackerGreen (left). FIG. 9F shows a representative example of a COS-7 cell inwhich both the IMM (cyan) and ER (magenta) were labeled with MitoTrackerGreen and Sec61β-mCherry and observed under 2D-SIM (upper left) andSparse-SIM (lower right). Contact sites of ER-mitochondria atmitochondrial cristae or matrix regions may be shown in the left lowerinset. FIG. 9G shows magnified views from the white box 930 in FIG. 9Fseen under Sparse-SIM, which showed the rearranged orientations (pointedby arrows 940-970) of inner cristae structures due to their dynamiccontacts with the ER (shown at left), while regions of rearrangedcristae from the white boxes are magnified (shown at right). Centerline, medians; limits, 75% and 25%; whiskers, maximum and minimum; errorbars, s.e.m; scale bars: (FIGS. 9A and F) 1 μm; (FIG. 9C) axial: 0.2arbitrary units (a.u.); lateral: 100 nm; (FIGS. 9B, 9E and 9G) 500 nm.

FIGS. 10A-10L illustrate exemplary results indicating sparse SD-SIMenables three-dimensional, multicolor and sub-90-nm resolution for livecell SR imaging according to some embodiments of the present disclosure.FIG. 10A shows a representative snapshot (the third frame, 0.2 sexposure per frame) of a COS-7 cell expressing clathrin-EGFP observedunder SD-SIM (left triangle) and Sparse SD-SIM (right triangle) with asCOMS camera. FIG. 10B shows the fluorescence intensity profilecorresponding to the resolved central ring of the CCP (upper left). FIG.10C shows average minimal resolutions of CCPs observed under SD-SIM andSparse SD-SIM as evaluated by the FRC technique. FIG. 10D showshistogram of the diameters of CCPs in live cells. FIG. 10E showsmagnified view of the boxed region 1010 in FIG. 4A, which shows thesequential disappearance of a CCP (indicated by the yellow arrows1030-1060). The top image may be a raw SD-SIM image; the rest may beSparse-SD-SIM images. FIG. 10F shows magnified view of the boxed region1020 in FIG. 10A, which shows the disintegration of the CCP into twoCCPs. FIG. 10G shows temporal projections of the colors of CCPs duringlive cell SR imaging for 16.75 minutes. FIG. 10H shows a representativeexample of four-color, live cell SR imaging by Sparse SD-SIM. The COS-7cells may be simultaneously labeled with LAMP1-mCherry (yellow),MitoTracker (green), Hoechst (blue), and Tubulin-EGFP (magenta). Asshown in FIG. 101, the magnified view of the boxed region 1070 in FIG.10H may be separated into two images highlighting mitochondria andlysosomes (left) and nucleus and tubulin (right). Views under the SD-SIMand the Sparse SD-SIM may be shown at the left and right of each image,respectively. FIG. 10J shows average resolutions of organelles labeledwith different colors observed under SD-SIM and Sparse SD-SIM andevaluated with the FRC technique. FIG. 10K shows three-dimensionaldistributions of all mitochondria (labeled with TOM20-mCherry) within avolume of 7 μm in the axial axis of a live COS-7 cell observed withSparse SD-SIM. Different colors may be used to code the differentdistances of the plane imaged from the cell surface. FIG. 10L showscolor-coded horizontal (left) and vertical sections (right) from theboxed region 1080 in FIG. 10K, in which images obtained with SD-SIM andSparse SD-SIM may be shown at the top and bottom, respectively. Centerline, medians; limits, 75% and 25%; whiskers, maximum and minimum; errorbars, s.e.m; scale bars: (FIGS. 10A and 10I) 3 μm; (FIGS. 10E and 10F)300 nm; (FIGS. 10G and 10L) 1 μm; (FIGS. 10H and 10K) 5 μm.

FIGS. 11A-11K illustrate exemplary results indicating that upsamplingmay enable Sparse SD-SIM to overcome the Nyquist sampling limit toaccomplish multicolor 3D SR imaging in live cells according to someembodiments of the present disclosure. FIG. 11A shows (from left toright) ER tubules in the periphery of a COS-7 cell (labeled withSec61β-EGFP) seen under the SD-SIM, Hessian SD-SIM, Sparse SD-SIM, andSparse SD-SIM×2 configurations. The SD-SIM system may be equipped withan EMCCD camera, whereas the physical pixel size in each SR image may besubstantially 94 nm. FIG. 11B shows average minimal resolutions ofimages obtained under different configurations from FIG. 11A evaluatedby the FRC technique. The theoretical resolution limit according to theNyquist criteria may be plotted as the dashed line 1120. FIG. 11C showsmagnified views of ER tubules from FIG. 11A and observed under differentconfigurations (from left to right). The ER tubular structures indicatedby arrows may be resolved only under the Sparse SD-SIM×2 configuration(the right inset). FIG. 11D shows Fourier transform of images obtainedunder the SD-SIM (left), the Hessian SD-SIM (middle left), the SparseSD-SIM (middle right), and the Sparse SD-SIM ×2 (right) configurations.FIG. 11E shows a snapshot of a HeLa cell labeled with tubulin-EGFP(magenta), Pex11a-BFP (cyan) and Lamp1-mCherry (yellow) observed underthe SD-SIM (left) and the Sparse SD-SIM×2 (right) configurations. FIG.11F shows magnified views of the solid boxed region 1130 in FIG. 11E, inwhich montages at different times (upper) show a lysosome moving alongthe microtubule to contact two peroxisomes. The contact between thelysosome and the peroxisomes may be further magnified and be shown atthe bottom. FIG. 11G shows time-lapse images of another example of theco-movement of both a lysosome and a peroxisome along a microtubuleunder the Sparse SD-SIM×2 configuration. The dashed lines may indicatethe microtubules. FIG. 11H shows live-cell three-color 3D imaging by theSparse SD-SIM ×2 approach. COS-7 cells may be labeled with Tubulin-EGFP(green), Hoechst (cyan), and MitoTracker Deep Red (magenta). FIG. 11Ishows the z-axial view from FIG. 11H recorded with SD-SIM (top) andSparse SD-SIM×2 (bottom). The fluorescent profiles of lines 1140(SD-SIM) and 1150 (Sparse SD-SIM×2) may across the microtubule filamentin the z-axis, which may be fitted with a Gaussian fitting, yielded FWHMresolutions of substantially 465 nm and substantially 228 nm for SD-SIMand Sparse SD-SIM×2, respectively. FIG. 11J shows three horizontalsections of the cellular nucleus (top) and mitochondria merged withmicrotubules (bottom), in which the inward and concave regions of thenucleus may be intruded by extensive microtubule filaments along withmitochondria. FIG.11K shows volumes of nuclei, mitochondria, andmicrotubules from left to right. The different imaging plane depths maybe coded in different colors. Scale bars: (FIGS. 11A, 11D, 11E, 11H,11I, 11J, and 11K) 5 μm; (FIGS. 11B and 11F top, and 11G) 3 μm; (FIG.11C) 1 μm; (FIG. 11F bottom) 500 nm.

FIG. 12 illustrates exemplary results indicating the detailed workflowof the sparse deconvolution according to some embodiments of the presentdisclosure. I-II, III-IV, and V-VI may be captured by 2D-SIM using thesCMOS camera, SD-SIM using the sCMOS camera, and SD-SIM using the EMCCDcamera, respectively. Then, the SR images may be reconstructed usinginverse Wiener filtering. Thereafter, overall similar but slightlydifferent strategies may be used to process these different types ofdatasets.

FIGS. 13A-13D illustrate exemplary results indicating benchmark ofspatial resolution at different steps of sparse deconvolution accordingto the synthetic images according to some embodiments of the presentdisclosure. FIG. 13A shows the resolution plate with a pixel size of16.25 nm, which may contain five pairs of lines at distances of 48.75nm, 65.00 nm, 81.25 nm, 97.50 nm, and 113.75 nm. The synthetic image(512×512 pixels) may be first illuminated by pattern excitation and thenconvoluted with a microscope PSF of (1.4 NA, 488 nm excitation). Thesignal may be recorded with an sCMOS camera with a pixel size of 65 nm,which may mean 4×downsampling of the original image (128×128 pixels).Gaussian noise with a variance of 5% of the peak intensity of the linemay be also included to the raw image. Then, (from left to right)inverse Wiener filtering may be used to obtain a conventional SIM image(256×256 pixels), followed by the reconstruction constrained bycontinuity and sparsity priori information and final deconvolution. Thetheoretical limit resolution of Wiener SIM may be calculated to be 97.6nm by following the equation λ/2(NA_(i)+NA_(d)), in which i and drespectively represent the illumination and detection NA. FIG. 13B showsthe corresponding intensities across different pairs of lines in FIG.13A here. Two lines separated by 65 nm may be resolved only when the rawimage underwent the full sparse deconvolution pipeline. As shown in FIG.13C, the SIM image obtained after Wiener inverse filtering to 512×512pixels may also be upsampled and processed it with other steps in thesparse deconvolution pipeline thereafter. FIG. 13D shows thecorresponding intensities across different pairs of lines in FIG. 13Chere. Note that two lines 48.75 nm apart may be separated by sparsedeconvolution in the upsampled image. Scale bars: 1 μm.

FIGS. 14A-14D illustrate exemplary results indicating that contributionsof different steps in sparse deconvolution of synthetic images maycorrupt with different noise extents according to some embodiments ofthe present disclosure. As shown in FIG. 14A, synthetic filamentstructures may be created using the Random Walk process and adopt themaximal curvatures in the program, generating the filament structures.To simulate time-lapse sequences of filament structures, a subpixelshift operation may be used to generate a random shift in thesefilaments based on the Markov chain in the t-axis, resulting in an imagestack of 128×128×5 pixels. To mimic SIM acquisition, these ground truthobjects may be illuminated by pattern excitation and convolute with amicroscope PSF of (1.4 NA, 488 nm excitation, 32.5 nm pixel) to generatewide-field raw images. In addition to the filaments, a background signalthat combines the effects of cellular autofluorescence and fluorescenceemitted from the out-of-focus plane (simulated by the convolution ofsynthetic filaments with out-of-focus PSF 1 μm away from the focalplane) may also be included. Moreover, Gaussian noise with differentvariance extents (0%, 11%, 25%, 50%, 80%) may be incorporated to thepeak fluorescence intensity of the filaments. The raw images may beacquired by an sCMOS camera with a pixel size of 65 nm and pixelamplitudes of 16 bits. FIG. 14B shows performance comparisons among fivedifferent types of SR reconstructions at 0% 11% and 80% noise levels(from top to bottom). Specifically, the raw images may be reconstructedwith inverse Wiener filtering (column 1), followed bycontinuity-constraint reconstruction only (column 2),continuity-constraint reconstruction and deconvolution (column 3),sparsity-constraint reconstruction only and deconvolution (column 4), orsparsity-plus-continuity-constraint reconstruction and deconvolution(column 5). FIG. 14C shows the fluorescence intensity profiles along thetwo opposing synthetic filaments in FIG. 14B under different conditions.While the continuity reconstruction followed by the deconvolution maynot separate these filaments, the sparsity reconstruction followed bythe deconvolution may cause artifacts (an example indicated by theasterisk) in raw images with 80% noise. As shown in FIG. 14D, comparedto the synthetic ground truth, the average peak signal-to-noise ratio(PSNR) and structural similarity (SSIM) values of SR imagesreconstructed with different techniques from raw images may corrupt withdifferent levels of noise (0%, 11, 25%, 50%, 80%). More details may begiven in Tables S1 and S2. Scale bars: 1 μm in FIG. 14A; axial: 0.1arbitrary units (a.u.) in FIG. 14C; lateral: 100 nm.

FIGS. 15A-15F illustrate exemplary results indicating bona fideextension of spatial resolution by the sparse deconvolution algorithmwhen processing real biological structures according to some embodimentsof the present disclosure. FIGS. 15A-15D show comparison betweenwide-field high NA (1.4) and low NA (0.8) imaging after the sparsedeconvolution process in both the spatial and Fourier spaces. As shownin FIG. 15A, from left to right may be the high NA (1. 4) image, thesimulated low NA (0. 8) image, and the low NA (0. 8) image after sparsedeconvolution (same data as in FIG. 8A). Here, the low NA image may begenerated using the formula iFFT{FFT(IMG_(1.4))×(OTF_(0.8)/OTF_(1.4))}.In the formula, the FFT (iFFT) denotes the (inverse) fast Fouriertransform, and IMG_(1.4) represents the high NA image. FIG. 15B showsthe optical transform functions of NA=1.4 (OTF_(1.4)) and NA=0.8(OTF_(0.8)). FIG. 15C shows the corresponding Gaussian-fitted crosssections of the insets in FIG. 15A. It may be clear that the two peakswith a distance of 259 nm may be resolved under the low NA after sparsedeconvolution, which is similar to 265 nm in the original high NA image.This may indicate that sparse deconvolution may achieve high NAresolution with better contrast. As shown in FIG. 15D, correspondingFourier space images may be presented from left to right. As shown inFIGS. 15E and 15F, a bona fide increase in spatial resolution may occurby the Sparse deconvolution of wide-field (WF) images according to theSIM images used as the ground truth. Live COS-7 cells overexpressingLifeAct-EGFP (FIG. 15E) or Clathrin-EGFP (FIG. 15F) may be used and beimaged with 2D-SIM, and TIRF-SIM respectively. From the raw dataset,regular deconvolution, continuity reconstruction followed bydeconvolution, upsampling plus sparsity and continuity reconstruction,or upsampling plus sparsity and continuity reconstruction followed bydeconvolution may be used. Clearly, only the full sparse deconvolutionalgorithm may be able to resolve the two opposing actin filaments (FIG.15E) and the ring structure of CCPs (FIG. 15F), the fidelity of whichwas confirmed by the SIM images on the right. Scale bars: (FIGS. 15A,15B, and 15D, and 15F) 500 nm; (A, inset) 200 nm.

FIGS. 16A and 16B illustrate exemplary results indicating OTFs obtainedby the Fourier transform of fluorescent beads visualized under differentconditions according to some embodiments of the present disclosure. FIG.16A shows images of fluorescent beads (48 nm in diameter) underwide-field imaging: (left upper), wide-field followed by deconvolution(right upper), wide-field plus sparsity and continuity reconstruction(left bottom), and wide-field plus sparsity and continuityreconstruction followed by deconvolution (right bottom). Upper insets: Asingle bead observed under different conditions; Lower insets: thecorresponding Fourier transform of the image on the left. FIG. 16B showsthe amplitudes along the white lines 1610-1640 in the frequency domainfrom the low insets in FIG. 16A, while the two vertical dashed lines1650 and 1660 indicate the spatial frequency limits of the wide-fieldmicroscope. Scale bar: 5 μm; inset: 500 nm.

FIGS. 17A and 17B illustrate exemplary results indicatingreconstructions with only sparsity, only continuity, or both thesparsity and continuity priors according to some embodiments of thepresent disclosure. FIG. 17A shows actin filaments labeled withLifeAct-EGFP in a live COS-7 cell in FIG. 8A under 2D-SIM (left),followed by sparsity-constraint reconstruction and deconvolution (middleleft), continuity-constraint reconstruction and deconvolution (middleright), or both sparsity and continuity-constraint reconstruction anddeconvolution (right). FIG. 17B shows CCPs labeled by Clathrin-EGFPunder 2D-SIM (left), followed by sparsity-constraint reconstruction anddeconvolution (middle left), continuity-constraint reconstruction anddeconvolution (middle right), or both sparsity and continuity-constraintreconstruction and deconvolution (right). Scale bars: (FIG. 17A) 1 μm;(FIG. 17B) 300 nm.

FIGS. 18A-18E illustrate exemplary results indicating three-dimensionalimage stack of fluorescent beads under SD-SIM and Sparse SD-SIMaccording to some embodiments of the present disclosure. FIGS. 18A-18Bshow a maximum intensity projection (MIP) view (left) and a horizontalsection (right) of fluorescent beads (100 nm in diameter) recorded bySD-SIM (FIG. 18A) and after the sparse deconvolution (FIG. 18B),respectively. Insets in the left-lower corner show a magnified view ofthe same fluorescent bead under different reconstruction algorithms.FIG. 18C shows the corresponding Gaussian fitted profiles in (FIG. 18A,left-lower corner), which may indicate that the lateral resolutions ofSD-SIM and Sparse SD-SIMmay be 165 nm and 82 nm, respectively. FIG. 18Dshows magnified horizontal sections from the boxes 1810 and 1820 inFIGS. 18A and 18B in the left and right panels, while the SD-SIM imagemay be processed with a median filter to avoid a nonconverged fittingresult. As shown in FIG. 18E, gaussian functions may be used to fit theintensity profiles along the axial direction of the fluorescent bead inFIG. 18D, yielding axial resolutions of 484 nm and 266 nm for SD-SIM andSparse SD-SIM, respectively. Scale bars: (FIG. 18B) 4 μm, (FIG. 18B,inset) 100 nm and (FIG. 18D) 200 nm.

FIGS. 19A-19D. illustrate exemplary results indicating two-colorlive-cell imaging of clathrin and actin by Sparse SD-SIM according tosome embodiments of the present disclosure. FIGS. 19A and 19B show colortemporal projections of CCPs (FIG. 19A) and the actin filament network(FIG. 19B) recorded by SD-SIM (left) and Sparse SD-SIM (right) for 16.75minutes at 5 s intervals. FIG. 19C shows CCPs (cyan) and the corticalactin cytoskeleton (magenta) in a COS-7 cell captured by Sparse SD-SIM.FIG. 19D shows montages of the boxed region 1910 FIG. 19C at five timepoints at a magnified scale. The first image observed under SD-SIM mayappear at the top left corner for comparison. It may be observed that aCCP may dock stably at the junction of two actin filaments and thendisappear from the focal plane as these neighboring filaments merge.Scale bars: (FIGS. 19A-19C) 4 μm; (FIG. 19D) 500 nm.

FIGS. 20A and 20B illustrate exemplary results indicating ER-lysosomecontacts revealed by the sparse SD-SIM according to some embodiments ofthe present disclosure. FIG. 20A shows contacts between ER tubules(labeled by Sec61-EGFP, green) and lysosomes (labeled by LysotrackerDeepRed, magenta) visualized by Sparse SD-SIM in live COS-7 cells. FIG.20B shows time-lapse images of typical lysosome-ER contact dynamicsmagnified from the dashed-boxed region 2010 in FIG. 20A by SD-SIM (top)and Sparse SD-SIM (bottom). The ER tubules may move along with thelysosome, followed by the contraction of the tubular structure to apolygon surrounding the lysosome (indicated by white arrows) and thefinal disappearance of the polygon due to the retraction of the ERtubules. Scale bars: (FIG. 20A) 3 μm; (FIG. 20B) 2 μm.

TABLE S1 PSNR of reconstructions by different algorithms from imagescorrupted with different noise levels (FIG. 13A) with respect to thesynthetic ground truth (G representing the Gaussian noise) Without noiseG(11%) G(25%) G(50%) G(80%) Wiener-SIM 18.08 10.90 9.39 8.27 11.32Continuity 17.94 10.85 9.40 8.23 9.73 Continuity^(+deconv) 23.01 13.9311.77 10.68 13.22 Sparsity^(+deconv) 27.04 17.74 15.88 15.29 15.24Sparse-SIM 27.92 18.72 18.08 17.97 17.83

TABLE S2 SSIM of reconstructions by different algorithms from imagescorrupted with different noise levels (FIG. S2A) with respect to thesynthetic ground truth (G representing the Gaussian noise) Without noiseG(11%) G(25%) G(50%) G(80%) Wiener-SIM 0.5612 0.2352 0.1773 0.14170.1445 Continuity 0.5500 0.2303 0.1724 0.1311 0.1545Continuity^(+deconv) 0.8311 0.2886 0.1724 0.1790 0.2107Sparsity^(+deconv) 0.9430 0.5203 0.4178 0.2884 0.3191 Sparse-SIM 0.95700.6832 0.6431 0.6167 0.6251

TABLE S3 Imaging conditions used in different Figures IlluminationWavelength Exposure intensity Microscope FIG. Label (nm) (ms) (W/cm²)SIM FIG. 8A LifeAct-EGFP 488 20 × 9  2 FIG. 8J Caveolin-EGFP 488 7 × 914 LAMP1-EGFP 488 20 × 9  2 FIG. 8M LysoView 488 488 20 × 9  2 LipidSpot488 488 20 × 9  2 FIG. 8O VAMP2-pHluorin 488 0.2 × 9  187 FIG. 9AMitoTracker Green 488 7 × 9 14 Tom20-mScarlet 561 7 × 9 50 FIG. 9FMitoTracker Green 488 7 × 9 14 Sec61β-mCherry 561 7 × 9 100 FIG. 15BClathrin-EGFP 488 7 × 9 14 SD-SIM FIG. 10A Clathrin-EGFP 488 200 0.91FIGs. 10G Lifeact -EGFP 488 200 0.89 and 10 Clathrin-DsRed 561 300 12.11FIG. 10K Tom20-mCherry 561 200 8.84 Hoechst H1399 405 200 0.87 FIG. 10HTubulin-EGFP 488 200 0.91 Lamp1-mCherry 561 200 8.84 MitoTracker ® DeepRed FM 640  50 1.07 FIG. 20 Sec61β-EGFP 488 200 0.77 Lysotracker DeepRed 640 200 0.92 FIG. 12 MitoTracker Green 488 150 0.43 FIG. 11ASec61β-EGFP 488 200 0.91 Pex11a-BFP 405 100 0.87 FIG. 11E Tubulin-EGFP488 300 0.91 Lamp1-mCherry 561 200 4.96 Hoechst H1399 405  50 0.87 FIG.11H Tubulin-EGFP 488 200 0.89 MitoTracker ® Deep Red FM 640  50 0.27

By developing the sparse deconvolution algorithm, the long-sought goalof applying mathematical SR to image data from different fluorescencemicroscopes may be accomplished. In the present disclosure, evidencethat demonstrates the fidelity of mathematical SR may be presented.First, in the simulated experiments with synthetic structures, it mayshow that sparse deconvolution may provide many more intricate detailsthat resemble the ground truth than conventional SIM deconvolution does(FIGS. 13 and 14). Second, by using sparse deconvolution on wide-fieldimages reconstructed from raw SIM images, it may be able to observeintricate structures inaccessible to wide-field microscopy, such asbranched filaments or the ring-shaped CCP (FIGS. 15E and 15F). Thefidelities of these structures may be fully confirmed by correspondingSR-SIM reconstructions. Third, for real biological structures withoutground truth images, the improvements in spatial resolution may bequantitatively evaluated by the FRC technique and the claimed resolutionmay be confirmed by its ability to distinguish adjacent structures. Inagreement with the nearly doubled resolution of SD-SIM fromsubstantially 150 nm to substantially 80 nm, it may be able to resolvemost ring-shaped CCPs whose mean diameters are centered at substantially150 nm (FIG. 9). Fourth, using the Sparse TIRF-SIM, it may be able toresolve many ring-shaped caveolae with diameters of substantially 60 nm.Likewise, the sizes of cortical actin mesh pores measured withSparse-SIM may be substantially 160 nm (FIG. 8I). Finally, when usingsparse deconvolution on an individual, a 2D image from the volumetricstack obtained by SD-SIM may enhance the axial resolution by twofold,while manually upsampling in combination with the sparse deconvolutionalgorithm may be able to retrieve the SR information formerly limited bythe Nyquist sampling criteria. Taken together, these data may suggestthat the spatial resolution extension by the sparse deconvolutionalgorithm may be legitimate and real and may be combined with otherhardware-based SR techniques to provide resolution enhancements with areduced photon budget (Table S3).

For microscopists, it may be long believed that the bandwidth of themicroscope may be invariant. However, an alternative point of view inwhich the total information carried by the microscope may be invariantmay be demonstrated by examples in the present disclosure. In thissense, introducing additional priori knowledge of the object beingimaged may be used to achieve mathematical SR. The bona fide extensionof the bandwidth frequency limited by the image acquisition device 110may be due to the deconvolution algorithm in the present disclosure. Theinclusion of the Hessian penalty on the object continuity may increaseimage SNR, which may be critical for any band-limited frequencyextrapolations beyond the OTF. Without the continuity prior, sparsedeconvolution may led to over-sharpening or random artifacts (FIGS. 14and 17). In contrast to the band-width extrapolation techniques, whichaim to reconstruct all the object details, the sparsity prior may beused as an adjustable parameter to constrain the iterative approximationof SR information. By adjusting the parameters to impose only a mildrequirement on the sparsity of objects in the final reconstructions, itmay avoid removing weak fluorescent signals and enable the algorithm tobe generally applicable to different datasets. The final iterativedeconvolution step may also essential to boost the high-frequencyinformation extended beyond the OTF and achieve the visual separation ofclosely packed structures (FIGS. 13, 14, 16, and 18). The fact that theresolutions of at least three types of microscopes may be extended bythe sparse deconvolution algorithm may suggest that the sparsedeconvolution algorithm may be generally applicable to other SRmicroscopy.

Among the different possible applications of sparse deconvolution,Sparse-SIM may achieve an unprecedented spatiotemporal resolutioncombination. At a substantially 60 nm resolution and a 564 Hz framerate, opening of vesicle fusion pores smaller than 90 nm may be seenearlier and longer pore durations may readily be observed (FIGS. 8O and8P), which may shed light on the molecular mechanism underlying vesiclefusion and other ultrafast processes. The different configurations ofouter and inner mitochondrial membranes observed by the Sparse-SIM mayindicate that it may also be useful in revealing details regardingorganelles and organelle contacts in live cells. For multicolorthree-dimensional SR imaging, Sparse SD-SIM may represent a betterchoice. At the price of reduced temporal resolution, Sparse SD-SIM maybetter preserve the resolution and contrast throughout different planeswithin the live cell, and the reduced illumination dosage may alsoenable long-term three-dimensional SR imaging.

In summary, it may be demonstrated the general applicability of thesparse deconvolution algorithm and its ability to mathematically enhancethe resolution of any existing fluorescence microscope images. Itscombination with hardware-based SR microscopes enables substantially 60nm resolution at unprecedented substantially millisecond (ms) temporalresolution and may enable routine multicolor, sub-90-nm SR imaging forbiologists, which may help the biologists better resolve differentstructures and dynamics in time and space in three dimensions.

Having thus described the basic concepts, it may be rather apparent tothose skilled in the art after reading this detailed disclosure that theforegoing detailed disclosure is intended to be presented by way ofexample only and is not limiting. Various alterations, improvements, andmodifications may occur and are intended to those skilled in the art,though not expressly stated herein. These alterations, improvements, andmodifications are intended to be suggested by this disclosure, and arewithin the spirit and scope of the exemplary embodiments of thisdisclosure.

Moreover, certain terminology has been used to describe embodiments ofthe present disclosure. For example, the terms “one embodiment,” “anembodiment,” and/or “some embodiments” mean that a particular feature,structure or characteristic described in connection with the embodimentis included in at least one embodiment of the present disclosure.Therefore, it is emphasized and should be appreciated that two or morereferences to “an embodiment” or “one embodiment” or “an alternativeembodiment” in various portions of this specification are notnecessarily all referring to the same embodiment. Furthermore, theparticular features, structures or characteristics may be combined assuitable in one or more embodiments of the present disclosure.

Further, it will be appreciated by one skilled in the art, aspects ofthe present disclosure may be illustrated and described herein in any ofa number of patentable classes or context including any new and usefulprocess, machine, manufacture, or composition of matter, or any new anduseful improvement thereof. Accordingly, aspects of the presentdisclosure may be implemented entirely hardware, entirely software(including firmware, resident software, micro-code, etc.) or combiningsoftware and hardware implementation that may all generally be referredto herein as a “unit,” “module,” or “system.” Furthermore, aspects ofthe present disclosure may take the form of a computer program productembodied in one or more computer readable media having computer readableprogram code embodied thereon.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including electro-magnetic, optical, or thelike, or any suitable combination thereof. A computer readable signalmedium may be any computer readable medium that is not a computerreadable storage medium and that may communicate, propagate, ortransport a program for use by or in connection with an instructionexecution system, apparatus, or device. Program code embodied on acomputer readable signal medium may be transmitted using any appropriatemedium, including wireless, wireline, optical fiber cable, RF, or thelike, or any suitable combination of the foregoing.

Computer program code for carrying out operations for aspects of thepresent disclosure may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB. NET,Python or the like, conventional procedural programming languages, suchas the “C” programming language, Visual Basic, Fortran 2103, Perl, COBOL2102, PHP, ABAP, dynamic programming languages such as Python, Ruby andGroovy, or other programming languages. The program code may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider) or in a cloud computing environment or offered as aservice such as a Software as a Service (SaaS).

Furthermore, the recited order of processing elements or sequences, orthe use of numbers, letters, or other designations therefore, is notintended to limit the claimed processes and methods to any order exceptas may be specified in the claims. Although the above disclosurediscusses through various examples what is currently considered to be avariety of useful embodiments of the disclosure, it is to be understoodthat such detail is solely for that purpose, and that the appendedclaims are not limited to the disclosed embodiments, but, on thecontrary, are intended to cover modifications and equivalentarrangements that are within the spirit and scope of the disclosedembodiments. For example, although the implementation of variouscomponents described above may be embodied in a hardware device, it mayalso be implemented as a software only solution, for example, aninstallation on an existing server or mobile device.

Similarly, it should be appreciated that in the foregoing description ofembodiments of the present disclosure, various features are sometimesgrouped together in a single embodiment, figure, or description thereoffor the purpose of streamlining the disclosure aiding in theunderstanding of one or more of the various inventive embodiments. Thismethod of disclosure, however, is not to be interpreted as reflecting anintention that the claimed subject matter requires more features thanare expressly recited in each claim. Rather, inventive embodiments liein less than all features of a single foregoing disclosed embodiment.

In some embodiments, the numbers expressing quantities or propertiesused to describe and claim certain embodiments of the application are tobe understood as being modified in some instances by the term “about,”“approximate,” or “substantially.” For example, “about,” “approximate,”or “substantially” may indicate ±20% variation of the value itdescribes, unless otherwise stated. Accordingly, in some embodiments,the numerical parameters set forth in the written description andattached claims are approximations that may vary depending upon thedesired properties sought to be obtained by a particular embodiment. Insome embodiments, the numerical parameters should be construed in lightof the number of reported significant digits and by applying ordinaryrounding techniques. Notwithstanding that the numerical ranges andparameters setting forth the broad scope of some embodiments of theapplication are approximations, the numerical values set forth in thespecific examples are reported as precisely as practicable.

Each of the patents, patent applications, publications of patentapplications, and other material, such as articles, books,specifications, publications, documents, things, and/or the like,referenced herein is hereby incorporated herein by this reference in itsentirety for all purposes, excepting any prosecution file historyassociated with same, any of same that is inconsistent with or inconflict with the present document, or any of same that may have alimiting affect as to the broadest scope of the claims now or laterassociated with the present document. By way of example, should there beany inconsistency or conflict between the descriptions, definition,and/or the use of a term associated with any of the incorporatedmaterial and that associated with the present document, the description,definition, and/or the use of the term in the present document shallprevail.

In closing, it is to be understood that the embodiments of theapplication disclosed herein are illustrative of the principles of theembodiments of the application. Other modifications that may be employedmay be within the scope of the application. Thus, by way of example, butnot of limitation, alternative configurations of the embodiments of theapplication may be utilized in accordance with the teachings herein.Accordingly, embodiments of the present application are not limited tothat precisely as shown and described.

1. A method for image processing, implemented on at least one machineeach of which has at least one processor and at least one storagedevice, comprising: generating a preliminary image by filtering imagedata generated by an image acquisition device; generating anintermediate image by performing, based on a first objective function, afirst iterative operation on the preliminary image, the first objectivefunction including a first term, a second term and a third term, thefirst term being associated with a first difference between theintermediate image and the preliminary image, the second term beingassociated with continuity of the intermediate image, the third termbeing associated with sparsity of the intermediate image; and generatinga target image by performing, based on a second objective function, asecond iterative operation on the intermediate image, the secondobjective function being associated with a system matrix of the imageacquisition device and a second difference between the intermediateimage and the target image.
 2. The method of claim 1, wherein thefiltering the image data comprises: filtering the image data byperforming Wiener inverse filtering on the image data.
 3. The method ofclaim 1, wherein the first iterative operation is an operation forminimizing the first objective function.
 4. The method of claim 1,wherein the first term includes a first weight factor relating to imagefidelity of the intermediate image or an L-2 norm of the firstdifference.
 5. (canceled)
 6. The method of claim 1, wherein the secondterm includes a Hessian matrix of the intermediate image.
 7. The methodof claim 1, wherein the third term includes a second weight factorrelating to the sparsity of the intermediate image or an L-1 norm of theintermediate image.
 8. (canceled)
 9. The method of claim 1, wherein thesecond iterative operation is an operation for minimizing the secondobjective function.
 10. The method of claim 1, wherein the seconditerative operation includes an iterative deconvolution.
 11. The methodof claim 1, wherein the second difference includes a third differencebetween the intermediate image and the target image multiplied by thesystem matrix.
 12. The method of claim 11, wherein the second objectivefunction includes an L-2 norm of the second difference.
 13. The methodof claim 1, further comprising: generating an estimated background imageby performing an iterative wavelet transformation operation on thepreliminary image.
 14. The method of claim 13, wherein the first term isfurther associated with a fourth difference between the first differenceand the estimated background image.
 15. (canceled)
 16. The method ofclaim 13, wherein the iterative wavelet transformation operationincludes one or more iterations, and each current iteration comprises:determining an input image based on the preliminary image or anestimated image generated in a previous iteration; generating adecomposed image by performing a multilevel wavelet decompositionoperation on the input image; generating a transformed image byperforming an inverse wavelet transformation operation on the decomposedimage; generating an updated transformed image based on the transformedimage and the input image; generating an estimated image for the currentiteration by performing a cut-off operation on the updated transformedimage; and in response to determining that a termination condition issatisfied, designating the estimated image as the background image. 17.The method of claim 1, wherein the first iterative operation comprises:determining an initial estimated intermediate image based on thepreliminary image; and updating an estimated intermediate image byperforming, based on the initial estimated intermediate image, aplurality of iterations of the first objective function, wherein in eachof the plurality of iterations, in response to determining that atermination condition is satisfied, finalizing the intermediate image.18. The method of claim 17, further comprising: performing an upsamplingoperation on the initial estimated intermediate image.
 19. The method ofclaim 1, wherein the target image has an improved spatial resolution incomparison with a reference image generated based on a third objectivefunction not including the third term of the first objective function.20. The method of claim 19, wherein a first spatial resolution of thetarget image is improved by 50%-100% with respect to a second spatialresolution of the reference image.
 21. The method of claim 1, whereinthe image acquisition device includes a structured illuminationmicroscope or a fluorescence microscope.
 22. (canceled)
 23. A system forimage processing, comprising: at least one storage medium including aset of instructions; and at least one processor in communication withthe at least one storage medium, wherein when executing the set ofinstructions, the at least one processor is directed to cause the systemto perform operations including: generating a preliminary image byfiltering image data generated by an image acquisition device;generating an intermediate image by performing, based on a firstobjective function, a first iterative operation on the preliminaryimage, the first objective function including a first term, a secondterm and a third term, the first term being associated with a firstdifference between the intermediate image and the preliminary image, thesecond term being associated with continuity of the intermediate image,the third term being associated with sparsity of the intermediate image;and generating a target image by performing, based on a second objectivefunction, a second iterative operation on the intermediate image, thesecond objective function being associated with a system matrix of theimage acquisition device and a second difference between theintermediate image and the target image.
 24. A non-transitory computerreadable medium, comprising at least one set of instructions for imageprocessing, wherein when executed by one or more processors of acomputing device, the at least one set of instructions causes thecomputing device to perform a method, the method comprising: generatinga preliminary image by filtering image data generated by an imageacquisition device; generating an intermediate image by performing,based on a first objective function, a first iterative operation on thepreliminary image, the first objective function including a first term,a second term and a third term, the first term being associated with afirst difference between the intermediate image and the preliminaryimage, the second term being associated with continuity of theintermediate image, the third term being associated with sparsity of theintermediate image; and generating a target image by performing, basedon a second objective function, a second iterative operation on theintermediate image, the second objective function being associated witha system matrix of the image acquisition device and a second differencebetween the intermediate image and the target image.
 25. (canceled)